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Preface 


The  purpose  of  this  study  was  to  develop  a  descriptive, 
empirical  report  of  the  relationships  among  quality  assur¬ 
ance  process  controls,  productivity  and  weapon  system  costs. 
This  purpose  was  selected  because  the  available  literature 
indicated  a  growing  interest  in  such  relationships,  but 
provided  little  empirical  evidence  to  support  claims  about 
such  relationships. 

Because  of  various  limitations,  this  study  could  only 
explore  the  relationships  that  exist  at  major  DoD  aerospace 
facilities.  This  exploration  was  done  by  way  of  a  detailed 
interview  of  key  government  personnel  stationed  at  randomly 
selected  facilities.  The  findings  reported  in  this  paper 
indicate  the  claimed  relationships  may  have  basis  in  fact, 
but  the  present  DoD  surveillance  methods  do  not  facilitate 
validation  of  the  contractor  data.  Although  these  results 
were  inconclusive,  this  topic  shows  promise  for  resolving 
the  concerns  various  interest  groups  have  regarding  the 
control  of  weapon  system  costs. 

In  conducting  this  study,  I  received  a  great  deal  of 
support  from  others.  I  am  most  indebted  to  my  faculty 
advisor,  Lt  Col  Terry  Clark,  for  his  patience,  guidance  and 
continuing  support.  I  also  wish  to  thank  my  reader,  Maj  Art 


Rastetter,  for  his  assistance  with  the  style  and  format 
nuances  of  military  writing.  A  word  of  thanks  is  also  owed 
those  unnamed  DoD  personnel  who  participated  in  the  on-site 
interviews  and  were  most  helpful  in  understanding  the  com¬ 
plexities  of  their  weapon  systems.  Finally,  I  wish  to  thank 
my  wife  Mary  and  our  three  wonderful  children  for  their 
patience,  support  and  understanding  during  my  extended  ab¬ 
sences  from  home  while  pursuing  my  degree. 


Michael  W.  O'Meara 
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Abstract 

X,  tWs 

^  This^otudy  was  a  preliminary  evaluation  of  the  rela¬ 
tionships  among  quality  assurance  process  controls,  produc¬ 
tivity  and  weapon  system  costs.  The  available  literature 
indicated  positive  relationships  should  exist  among  the 
three  elements  examined,  but  little  empirical  evidence  was 
presented  to  support  the  positions  of  the  authors.  As  a 
result,  a  detailed  interview  methodol-o-gy  was  developed  to 
describe  the  relationships  that  exist  at  major  DoD  aerospace 
facilities.  On-site  interviews  were  conducted  at  five  ran¬ 
domly  selected  aerospace  facilities  identified  as  DoD  Plant 
Representative  Offices.  The  results  of  this  initial  study 
were  inconclusive.  The  examined  relationships  appeared  to 
be  positive,  but  the  cognizant  DoD  personnel  had  not  vali¬ 
dated  the  contractor  data  claiming  increased  productivity 
and  reduced  costs  associated  with  adequate  process  controls. 
Recommendations  are  provided  to  improve  the  DoD  surveillance 
methods*,  and  possible  follow-on  research  topics  are  identi- 


AM  AMALTSIS  OF  THE  EFFECT  OF  PROCESS  CONTROLS  ON 


PRODUCTIVITY  AND  WEAPON  SYSTEM  COSTS  IN  DOD  PROCUREMENT 


I •  Introduction 


Justification 

This  paper  reports  the  results  of  a  graduate-level 
research  project  evaluating  the  relationships  among  quality 
assurance  process  controls,  productivity  and  weapon  system 
costs.  The  research  objectives  described  in  this  chapter 
may  have  a  direct  ^impact  on  the  Department  of  Defense's 
efforts  to  reduce  weapon  system  costs.  The  investigative 
questions  selected  to  accomplish  the  research  objectives 
have  generally  not  been  empirically  tested  by  prior  re¬ 
search.  As  a  result,  this  study  will  determine  whether  the 
identified  relationship  between  quality  assurance  process 
controls  and  productivity  exists,  and  ultimately,  whether  a 
relationship  can  be  demonstrated  between  improved  productiv¬ 
ity  and  reduced  weapon  system  costs. 
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General  Issue 


In  April  1981,  former  Deputy  Secretary  of  Defense  Frank 
Carlucci  (5:p. 9-140)  issued  a  memorandum  about  improving  the 
acquisition  process  for  the  Department  of  Defense  (DoD). 
Mr.  Carlucci  wrote,  "A  primary  objective  in  streamlining  the 
acquisition  process  is  reducing  costs."  He  was  determined  to 
reduce  cost  overruns  in  DoD  acquisitions. 

Efforts  are  being  made  by  the  military  services  to 
identify  methods  to  reduce  the  cost  of  major  weapon  systems. 
Productivity  and  quality  improvement  programs  have  been 
referred  to  in  the  literature  (14:Q2;  24:Q2-Q6;  25:15; 
31:67)  as  one  approach  to  accomplish  the  goal  of  reduced 
product  costs. 

Problem  Statement 

The  issue  identified  in  the  preceding  paragraph  could 
generate  a  multitude  of  studies.  In  narrowing  the  topic  for 
this  research  project,  two  specific  research  questions  were 
developed  from  the  general  issue: 

1.  Are  DoD  in-plant  personnel  aware  of  a  relationship 
between  quality  assurance  process  controls  and  produc¬ 
tivity? 

2.  If  such  a  relationship  exists,  are  associated  cost 
savings  passed  on  to  DoD  in  future  weapon  system  pro¬ 
curements? 


Investigative  Questions 

To  evaluate  the  research  questions  identified  in  the 
preceding  section,  the  following  investigative  questions 
were  tested  during  this  study: 

1.  How  do  DoD  in-plant  personnel  evaluate  contractor 
control  of  production  and  manufacturing  prc  **sses? 

2.  How  do  DoD  in-plant  personnel  evaluate  the  contrac¬ 
tor's  cost  controls? 

3.  If  the  DoD  in-plant  personnel  evaluate  the  process 
controls,  how  do  they  define  a  relationship  between 
such  controls  and  productivity  improvement  or  cost? 

4.  What  data  is  available  to  demonstrate  relationships 
among  process  controls,  productivity  improvement  and 
reduced  costs? 

Research  Ob lectlves 

The  preceding  investigative  questions  were  designed  to 
measure  the  extent  of  DoD  awareness  regarding  contractor 
quality  assurance  controls  over  production  and  manufacturing 
processes.  The  ultimate  objective  of  this  study  was  an 
evaluation  of  the  relationship  of  such  controls  to  produc¬ 


tivity  and  cost.  A  secondary  evaluation  addressed  the  issue 
of  whether  DoD  in-plant  personnel  use  their  knowledge  of 
these  relationships,  if  they  exist,  to  assure  reduced  pro¬ 
curement  costs  for  DoD  weapon  systems. 

According  to  Emory,  some  of  the  typical  objectives  of  a 
research  study  include: 

(1)  descriptions  of  phenomena  or  characteristics  asso¬ 
ciated  with  a  subject  population,  (2)  estimates  of  the 
proportions  of  a  population  that  have  these  character¬ 
istics,  [and]  (3)  discovery  of  associations  among  dif¬ 
ferent  variables  .  .  .  [13:91]. 


The  research  objectives  of  this  study  provided  an  opportun¬ 
ity  to  test  the  criteria  identified  by  Emory.  As  noted  at 
the  beginning  of  this  chapter,  empirical  testing  of  this 
study's  investigative  questions  has  not  been  reported  in  the 
literature.  As  a  result,  the  research  objectives  of  this 
report  were  accomplished  by  means  of  a  descriptive  study. 
Emory  (13:91)  noted  that  such  studies  provide  the  opportun¬ 
ity  to  learn  the  who,  what,  when,  where  and  how  of  a  topic. 

Overview  of  the  Research  Tonic 

In  1981,  former  Deputy  Secretary  of  Defense  Frank  Car- 
lucci  issued  32  initiatives  for  improving  the  DoD  acquisi¬ 
tion  process.  One  of  the  major  objectives  of  these  initia¬ 
tives  was  to  reduce  the  cost  of  major  weapon  system  acquisi¬ 
tion  (5:p. 9-138).  One  of  the  specific  decisions  (recommen¬ 
dation  5)  supporting  this  objective  was  "to  encourage  capi¬ 
tal  investment  to  enhance  productivity  [  5 :  p.9-1 5 1 ] ."  The 
documentation  supporting  this  decision  cited  the  low  profit 
associated  with  DoD  contracting  as  one  contributor  to  the 
low  rates  of  capital  investment  (5:p. 9-151). 

The  Carlucci  initiative  identified  in  the  preceding 
paragraph  tied  contractor  profits  to  increased  productivity, 
and  increased  productivity  to  reduced  weapon  system  costs. 
The  current  literature  contains  a  number  of  citations 
(14:Q2;  24:Q2-Q6;  25:15;  31:67)  claiming  a  relationship 


between  effective  quality  assurance,  productivity  improve¬ 
ment,  and  product  or  weapon  system  costs.  These  relation¬ 
ships  will  be  described  in  detail  in  the  literature  review 
contained  in  Chapter  II. 

MIL-Q-9858A,  Quality  Program  Requirements,  issued  in 
1963,  recognized  the  relationship  between  quality  and  cost 
(39:1)  and  required  that  the  "[quality]  program  shall  fac¬ 
ilitate  determination  of  the  effects  of  quality  deficiencies 
and  quality  costs  on  price."  MIL-Q-9858A  also  described 
the  essential  elements  for  assuring  product  quality.  One  of 
the  key  elements  listed  in  MIL-Q-9858A  is  production  (or 
process)  control  (39:6).  Process  control  is  also  frequently 
cited  (24:Q8;  25:15;  29:Q2;  40:44)  in  the  literature  as  a 
critical  element  in  productivity  improvement  programs. 

In  addition  to  the  DoD  concerns  brought  out  in  the 
Carlucci  initiative,  one  general  concern  voiced  in  much  of 
the  literature  (6:1;  8:36;  19:16;  25:14;  28:14;  40:42)  was 
the  recognized  difference  between  the  quality  level  of  Jap¬ 
anese  products  and  the  quality  level  of  American  products. 
Just  like  MIL-Q-9858A,  evaluations  of  Japanese  quality  pro¬ 
grams  (6:4-6;  25:14-15;  40:42-43)  have  often  cited  produc¬ 
tion  (or  process)  control  as  a  key  element  of  product  quali¬ 
ty.  The  wide  recognition  of  the  value  of  this  control 
element  and  its  effect  on  product  costs  are  described  in 
detail  in  Chapter  II  of  this  report. 


The  references  cited  in  this  report  are  not  peculiar  to 
DoD;  instead,  they  present  industrial,  quality  consultant 
and  military  issues  and  recommendations.  This  research 
project  investigated  whether  these  general  issues  and  recom¬ 
mendations  can  ultimately  be  used  to  reduce  weapon  system 
costs. 


Definitions 


Productivity.  Tuttle  (32:7)  stated,  "Defining  pro¬ 
ductivity  is  particularly  challenging  because  the  term  has 
been  used  in  many  different  ways."  He  then  described  pro¬ 
ductivity  from  a  number  of  different  perspectives.  For  the 
purpose  of  this  report,  the  following  elements  are  consi¬ 
dered  essential  to  defining  productivity  (32:7,11): 

Productivity  is  the  relationship  between  output 
and  its  associated  inputs  when  the  outputs  and  Inputs 
are  expressed  in  real  (physical  volume)  terms. 

While  economists  traditionally  define  productivity 
as  the  ratio  of  outputs  [to  inputs]  stated  in  real  or 
physical  terms,  they  sometimes  substitute  prices  or 
costs  for  physical  volumes. 

Most  .  .  .  managers  would  find  .  .  .  the  econo¬ 
mist's  .  .  .  definition  of  productivity  to  be  too 
narrow.  Virtually  9  out  of  10  managers  would  include 
quality  ...  in  their  definition. 

Quality.  As  defined  in  MIL-STD-109B  (38:section  3), 
quality  is  "the  composite  of  all  the  attributes  or  charact¬ 
eristics,  including  performance,  of  an  item  or  product." 

Quality  Assurance.  DoD  Directive  4155.1  (37:  enclosure 


2)  defined  quality  assurance  as: 


A  planned  and  systematic  pattern  of  all  actions  neces¬ 
sary  to  provide  adequate  confidence  that  materiel, 
data,  supplies  and  services  conform  to  established 
technical  requirements  and  achieve  satisfactory  perfor¬ 
mance. 

Yield.  Using  the  measures  most  frequently  cited  in  the 
literature  (7:38;  12:App  C.para  4. 8. 2. 3;  14:Q2;  26:48;  27:1; 
39:3),  the  yield  of  a  process  or  manufacturing  operation  is 
the  percentage  of  conforming  items  that  complete  the  process 
or  operation.  The  number  of  conforming  items  is  determined 
by  deducting  nonconformances  from  the  total  items  started  in 
the  process  or  operation.  Typical  nonconformances  cited 
include  items  that  are  scrapped,  items  that  are  reworked  to 
meet  specifications,  and  those  items  repaired  to  a  more  con¬ 
forming  state. 

Scope  and  Limitations 

There  are  many  possible  avenues  to  investigate  the  cost 
of  DoD  acquisitions.  This  study  evaluated  one  element  of  a 
quality  assurance  program  —  process  control  —  to  determine 
if  a  relationship  exists  between  it  and  productivity,  and 
ultimately  between  productivity  and  cost. 

The  topic  is  further  limited  by  the  methodology  de¬ 
scribed  in  Chapter  III.  The  first  limitation  occurred  by 
way  of  the  decision  to  perform  the  study  at  aerospace  con¬ 
tractor  facilities  known  as  plant  representative  offices  of 
the  Department  of  Defense.  Facilities  such  as  as  Air  Force 
Plant  Representative  Offices  (AFPROs),  Navy  Plant  Represen¬ 
tative  Offices  (NAVPROs),  Defense  Contract  Administrative 


7 


Services  Plant  Representative  Offices  (DCASPROs),  etc.  are 
frequently  referred  to  by  their  acronyms  in  the  aerospace 
community.  Chapter  III  also  addresses  time  and  money  con¬ 
straints  which  limited  the  number  of  facilities  visited,  and 
resulted  in  an  uncontrolled  sampling  error  associated  with 
the  data  from  this  study.  However,  the  impact  of  this 
limitation  was  attenuated  by  other  sampling  controls  de¬ 
scribed  in  Chapter  III,  so  that  acceptable  levels  of  data 
accuracy  were  ultimately  obtained. 

One  final  consideration  must  be  made  while  addressing 
the  scope  of  this  study.  While  this  thesis  study  is  spon¬ 
sored  by  the  Air  Force  Institute  of  Technology,  the  research 
questions  and  objectives  are  important  to  all  DoD  procure¬ 


ment  activities. 


Chapter  II  begins  with  a  review  of  the  literature 
relevant  to  this  study.  This  literature  review  describes  a 
relationship  between  productivity  and  reduced  costs,  and 
includes  process  control  as  the  primary  means  of  achieving 
the  desired  increases  in  productivity.  The  literature  re¬ 
view  concludes  with  two  subsections  that  report  gains  asso¬ 
ciated  with  productivity  improvement  programs  and  a  discus¬ 
sion  of  the  empirical  support  for  the  conclusions.  Chapter 
II  then  continues  with  a  section  containing  this  writer's 
analysis  of  the  literature,  and  concludes  with  a  section  on 
the  significance  of  the  findings  to  DoD. 


Chapter  III  is  a  detailed  description  of  the  survey 
methodology  employed  in  this  atudy.  After  a  short  overview 
of  the  chapter,  the  justification  for  the  selected  methodo¬ 
logy  is  presented.  This  section  is  followed  by  an  in-depth 
description  of  the  methodology  and  survey  instrument,  in¬ 
cluding  a  review  of  the  validity  and  reliability  of  the 
approach  selected.  Chapter  III  continues  with  a  section 
describing  the  study  population,  and  a  justification  of  the 
representativeness  of  the  sampling  technique  used  to  eval¬ 
uate  the  population.  The  data  collection  plan  and  survey 
variables  are  then  described,  followed  by  a  concluding  sec¬ 
tion  which  describes  the  planned  statistical  analysis  of  the 
sample  data. 

Chapter  IV  reports  the  findings  of  the  survey  and 
provides  an  in-depth  analysis  of  the  results.  Like  the 
preceding  chapter,  this  chapter  begins  with  a  short  over¬ 
view,  followed  by  a  section  describing  the  implementation  of 
the  survey  and  the  variations  from  the  planned  design.  The 
actual  results  of  the  study  are  then  reported,  together  with 
a  reduction  of  the  data  and  an  analysis  of  the  results  in 
terms  of  the  original  investigative  and  research  questions. 
Chapter  IV  concludes  with  a  report  of  the  statistical  ana¬ 
lyses  appropriate  for  the  data  generated  by  this  study. 


Chapter  V  contains  the  conclusions  and  recommendations 
of  this  writer  based  on  the  results  obtained  from  the  sur¬ 
vey.  After  a  short  overview  section,  this  chapter  describes 


the  significance  of  the  findings  in  terms  of  the  research 
questions  and  the  general  issue  described  earlier  in  this 
chapter.  A  third  section  then  describes  the  practical  im¬ 
plications  of  the  results  reported  in  Chapter  IV.  Chapter  V 
then  closes  this  report  with  a  section  containing  recommen¬ 
dations  for  follow-on  or  revised  studies  based  on  the  data 
generated  by  this  report. 


II.  Literature  Review 


Topical  Literature 

Introduction.  The  overview  section  in  the  previous 
chapter  mentioned  the  DoD  concern  for  enhancing  productivity 
(5:p. 9-151).  The  Carlucci  initiative  in  dealing  with  this 
concern  associated  increased  productivity  with  reduced  costs 
( 5 : p. 9-151),  and  contained  a  recommendation  to  enhance  pro¬ 
ductivity  through  increased  capital  investment.  The  Car¬ 
lucci  memorandum  identified  a  number  of  causes  for  the  low 
investment  in  capital  equipment.  The  memorandum  also  pro¬ 
vided  recommendations  to  overcome  these  shortcomings  by  such 
actions  as  changing  the  DoD  depreciation  methods,  and  sup¬ 
porting  the  DoD  Manufacturing  Technology  programs. 

General  Robert  T.  Marsh,  USAF,  as  the  Commander  of  the 
Air  Force  Systems  Command  (AFSC),  was  responsible  for  yearly 
weapon  system  procurements  worth  billions  of  dollars.  Gen¬ 
eral  Marsh  (22:62)  described  his  concern  for  weapon  system 
costs  at  a  1983  Defense  Logistics  Agency  conference  on 
quality.  This  concern  is  identical  to  the  concern  expressed 
by  Mr.  Carlucci  in  the  preceding  reference.  But,  General 
Marsh  proposed  a  different  methodology  for  addressing  their 
mutual  concern,  when  he  emphasized  the  need  for  improved 
quality.  General  Marsh  stated. 


A  rather  informal  look  at  Air  Force  contracts,  the  year 
before  last,  revealed  that  at  least  $570  million  was 
blown  away  by  scrap,  rework  and  repair,  funds  that 
never  would  have  been  wasted  had  quality  been  designed- 
in  from  the  start. 

The  National  Research  Council's  Committee  on  Navy  Ship¬ 
building  Technology  (7:p.v)  combined  the  Carlucci  and  Marsh 
methodologies  into  a  perspective  that  linked  quality  and 
productivity.  Like  Carlucci,  their  study  of  productivity 
changes  in  Navy  shipbuilding  recognized  the  utility  of  DoD’s 
Manufacturing  Technology  programs  in  improving  productivity 
as  well  as  quality. 

This  research  project  will  ultimately  address  the  com¬ 
bined  effects  of  quality  and  productivity  on  weapon  system 
costs.  An  evaluation  of  the  relevant  literature  to  deter¬ 
mine  if  a  relationship  exists  between  capital  equipment  and 
productivity  begins  on  the  next  page.  This  narrow  perspec¬ 
tive  is  then  broadened  by  reviewing  the  conclusions  of 
various  authors  who  describe  the  role  such  equipment  plays 
in  process  control  and  the  relationship  of  process  control 
to  quality.  Finally,  additional  elements  of  process  con¬ 
trol,  identified  by  the  various  authors,  will  be  considered 
to  determine  their  impact  on  quality  and  productivity.  All 
of  the  identified  elements  of  process  control  will  be  evalu¬ 
ated  in  the  research  methodology  described  in  Chapter  III. 
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Productivity  and  Capital  Equipment.  Like  Carlucci, 
Japanese  planners  believe  chere  is  a  relationship  between 
capital  investment,  depreciation,  tax  benefits  and  produc¬ 
tivity  (28:15).  A  Quality  magazine  interview  with  Mr.  Joji 
Aral,  Manager  of  the  Japan  Productivity  Center  (28:18), 
provided  a  comparison  of  the  Japanese  and  American  tax 
benefits  associated  with  depreciation  of  new  equipment.  In 
part,  Mr.  Arai  stated: 

Depending  on  the  type  of  investment,  you  can  have 
depreciation  of  up  to  14  percent  in  the  first  year  in 
excess  of  regular  depreciation.  Those  tax  systems  cer¬ 
tainly  help  the  Japanese  invest  more  [than  American 
companies]  in  the  plant,  new  equipment  and  especially 
in  research  and  development  [28:15]. 

T.  J.  Murrin,  President  of  the  Public  Systems  Company, 
Westinghouse  Electric  Corporation,  has  responsibility  for 
the  subsidiary  Defense  System  Center  contracts.  At  the  1982 
Annual  Quality  Congress,  hosted  by  the  American  Society  for 
Quality  Control  (ASQC),  Mr.  Murrin  described  his  company's 
efforts  to  improve  quality  and  productivity  (25:14).  Mr. 
Murrin  went  on  to  provide  some  insight  to  the  Westinghouse 
philosophy  regarding  new  equipment,  when  he  stated,  "Another 
strategy  is  the  introduction  of  improved  technology  to  let 
us  do  the  job  right  with  the  right  tools." 

The  U.S.  Air  Force  supports  Murrin's  beliefs  regarding 
the  role  of  new  capital  equipment.  In  the  Air  Force  gui¬ 
dance  to  their  Technology  Modernization  (Tech  Mod)  program, 
they  state: 


The  primary  objective  of  Tech  Mod  is  to  improve 
the  overall  health  of  the  industrial  base  through 
implementation  of  manufacturing  technology  and  improved 
capital  investment.  The  resulting  improvements  in 
productivity  generate  numerous  benefits  for  both  the 
Air  Force  and  the  private  sector,  including:  reduced 
costs;  improved  quality  .  .  .  [12:1-3]. 

Rear  Admiral  Frank  C.  Collins,  USN,  was  Executive  Dir¬ 
ector  of  Quality  for  the  Defense  Logistics  Agency  (DLA). 
Admiral  Collins  (6:9)  provided  a  different  perspective  re¬ 
garding  capital  equipment  when  he  described  DoD's  inability 
to  invest  in  new  equipment: 

Regrettably,  we  are  still  recycling  1950's 
technology  in  our  Industrial  Plant  Equipment  (IPE) 
program,  in  which  DLA  pays  for  installation  and 
maintenance  of  equipment  for  defense  industries  that 
thus  escape,  over  the  short  term,  the  tough  decisions 
to  retool  to  modern  technological  state  of  art. 


The  Role  of  Capital 


in  Process  Control.  Don 


Curtis,  the  manager  of  Hewlett-Packard's  Disc  Memory  Divi¬ 
sion,  echoed  Admiral  Collins'  views  and  also  tied  these 
efforts  to  cost  savings  (24:Q8).  Mr.  Curtis  continued  by 
describing  the  justification  process  for  new  equipment  used 
at  the  Hewlett-Packard  Disc  Memory  Division: 

One  of  the  key  philosophies  involves  automation. 
We  originally  justified  the  investment  in  automation 
merely  on  the  basis  of  how  fast  we  could  recover 
through  reduced  direct  labor  costs.  We  now  have  a 
list  of  eight  reasons  to  automate  production  --  and 
number  eight  is  justified  labor  .  .  .  the  second  one 
is  process  control  —  quality. 

The  use  of  capital  equipment  in  process  control  was  one 
of  the  objectives  of  the  Air  Force  Tech  Mod  program  cited 
above.  In  an  Air  Force  sponsored  study,  Gossard,  et  al 
(15:5-8)  described  the  effects  of  new  designs  and  techniques 


that  can  be  used  with  capital  equipment.  The  process  con¬ 
trol  technique  they  tested  was  described  as  adaptive  brake¬ 
forming  of  sheet  metal  parts.  They  found  some  of  the  poten¬ 
tial  benefits  of  this  method  of  process  control  were  in¬ 
creased  part  accuracy  and  improved  quality  control. 

Part  accuracy  is  often  measured  by  various  types  of 
capital  equipment.  The  advantage  of  using  automated  capital 
equipment  in  quality  control  testing  was  described  by  Craig 
Walter,  another  Hewlett-Packard  manager: 

The  automatic  test  system  also  obviates  the  need  for 
final  QA,  as  it  is  presently  known.  The  role  of  final 
QA  becomes  one  of  insuring  the  integrity  of  the  process 
instead  of  screening  the  product  [40:44]. 

Automation  is  not  without  its  drawbacks.  An  article  in 

Quality  magazine  reminded  those  who  see  automation  as  a 

panacea:  "Automating  an  operation  [or  process]  that  is  not 

in  control  assures  the  quick  and  consistent  manufacture  of 

defective  product  [29:Q2]." 

Defective  product,  as  will  be  illustrated  in  the  fol¬ 
lowing  subsections,  contributes  to  increased  costs.  As  a 
result  of  this  drawback,  this  study  will  evaluate  more  than 
just  the  role  of  automation  in  process  control.  The  follow¬ 
ing  subsections  describe  some  of  the  other  elements  cited  as 
essential  to  improved  process  control  and  increased  produc¬ 
tivity. 


An  Expanded  View  of  Process  Control.  The  last  eight 
citations  address  only  one  component  of  process  control  — 
capital  investment.  In  addition  to  this  component  identi¬ 
fied  in  the  Carlucci  initiative  (5:p. 9-151),  there  are  many 
other  facets  of  process  control.  For  example,  a  number  of 
sources  emphasize  the  necessity  of  understanding  the  spread 
or  variability  of  the  process  (3:27-86;  16:72,91; 
31:64,82,240;  40:43-44).  Statistical  analysis  is  essential 
at  this  point.  As  long  as  the  engineering  or  design  toler¬ 
ances  are  compatible  with  the  process  capability,  production 
yield  will  be  maximized. 

Broh  (4:53-59)  covered  other  facets  of  process  control 
when  he  emphasized  equipment  capability  must  be  monitored  to 
assure  accuracy  and  precision  are  maintained  throughout  the 
life  of  the  process.  He  also  noted  that  operating  environ¬ 
ment  and  maintenance  must  be  considered  by  the  quality 
planner  evaluating  a  process.  And  finally,  he  reported  that 
the  equipment  set-up  techniques  must  be  considered  when  the 
machine  is  used  for  manufacturing  more  than  one  product. 

Keith  McKee  (23:20),  Director  of  the  Manufacturing 
Productivity  Center  of  the  IIT  Research  Institute,  pointed 
out  the  need  to  monitor  the  condition  of  the  tooling  used 
with  the  manufacturing  equipment.  "By  monitoring  tool  wear 
and  position  .  .  .  companies  can  obtain  the  knowledge  they 
need  to  adjust  the  tools  or  stop  the  process  before  bad 
parts  are  made." 


Relating  Process  Controls  to  Productivity  and  Profits. 
The  preceding  citation  identified  the  ultimate  goal  of  im¬ 
proved  process  control  —  to  increase  productivity  by  assur¬ 
ing  the  manufacture  of  quality  parts.  Don  Curtis  (24:Q2-Q6) 
described  this  relationship  when  he  stated. 

Quality  is  the  maintenance  of  a  designed-in  level 
of  reliability  through  high  standards,  workmanship  and 
process  control.  Productivity  is  just  an  extension  of 
quality.  Quality  improvement  automatically  increases 
productivity  —  you  don’t  have  to  do  anything  else. 

Murrin  (25:15)  and  Squires  (31:67)  reversed  this  logic 
in  describing  the  absence  of  process  control  as  the  cause  of 
increased  cost.  Fiegenbaum  (14:Q2)  supported  this  position, 
illustrating  the  situation  that  exists  when  quality  improve¬ 
ments  are  not  evident: 

Industrial  practices  based  on  traditional  concepts  have 
created  a  "hidden  plant"  that  drains  as  much  as  40 
percent  of  an  operation's  productive  capacity.  This 
plant  exists  to  replace  products  recalled  from  the 
field,  to  retest  and  reinspect  rejected  units,  [or  to] 
rework  unsatisfactory  parts. 

The  Committee  on  Navy  Shipbuilding  Technology  identi¬ 
fied  a  similar  conclusion  regarding  the  effect  of  quality  on 
weapon  system  costs.  They  (7:38)  first  described  the  result 
of  improved  quality  performance,  which  they  believed  pre¬ 
cluded  the  need  for  rework  and  repair.  They  further  noted, 
"Rework  and  repair,  in  addition  to  adding  to  costs,  usually 
must  be  done  in  a  less  efficient  manner  than  was  done  in  the 
first  place." 
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The  last  two  citations  refer  to  elements  that  are 
included  in  this  study's  definition  of  yield.  These  same 
elements  are  found  in  MIL-Q-9858A  (39:3,7)  and  MIL-STD-1520A 
(11:5),  and  reflect  just  some  of  the  costs  or  losses  identi¬ 
fied  by  an  adequate  quality  program  (20:47).  Puma  (27:15) 
described  the  role  quality  data  can  play  in  determining  both 
quality  effectiveness  and  productivity.  Unfortunately,  MIL- 
Q-9858A  permits  the  contractor  to  determine  what  quality 
cost  data  will  be  maintained.  Vhile  this  factor  is  beyond 
the  scope  of  this  study,  it  could  lead  to  inconsistencies  in 
measuring  the  effectiveness  of  process  controls.  As  a  re¬ 
sult,  this  issue  will  be  addressed  as  one  of  the  proposed 
follow-on  studies  described  in  Chapter  V. 

Reported  Gains.  Although  the  preceding  remarks  have 
implied  a  relationship  between  quality  elements  and  produc¬ 
tivity,  the  approaches  of  most  of  the  authors  reviewed  to 
this  point  were  philosophical.  They  did  not  cite  the  meas¬ 
ure  of  productivity  used  to  support  their  conclusions. 
Their  reports  did  not  contain  empirical  evidence  to  support 
their  conclusions;  but,  as  noted  below,  some  did  report 
gains  associated  with  their  recommended  approach. 

Murrin  (26:47)  stated  the  Westinghouse  Public  Systems 
Company  "has  achieved  a  seven  percent  annual  productivity 
increase  over  the  past  three  years."  He  also  described  an 
example  of  how  his  company's  approach  was  implemented  on  a 


printed  wiring  assembly  program: 


Statistical  techniques  were  used  to  determine  the 
priority  of  each  problem  based  on  its  effect  on  yield. 
In  the  first  year,  the  number  of  wiring  assemblies  -- 
passing  all  process  steps  without  needing  any  rework 
—  has  climbed  dramatically.  And  productivity  is  up 
more  than  35  percent  [26:48]! 

Walter  (41:23-24)  described  the  effect  of  Hewlett- 
Packard's  quality  assurance  program  at  two  different  facil¬ 
ities.  At  the  Loveland  Instrument  Division,  "Wave  soldering 
nonconformances  were  reduced  from  a  few  tenths  of  a  percent 
to  200  ppm."  This  reduction  in  defects  translated  to 
increased  process  yield  which,  by  definition,  is  another 
measure  of  increased  productivity.  Walter  continued  to 
describe  the  Hewlett-Packard  improvements  by  citing  their 
success  at  Togahama/Hewlett-Packard: 

Production  costs  have  declined  by  more  than  a  third. 
Their  inventory  has  been  reduced  by  two-thirds  .  .  .. 
,  The  average  yield  failure  rate  of  our  products  has  been 
reduced  about  16%  per  year  [41:23-24], 

Empirical  Studies.  As  noted  in  the  preceding  sub¬ 
section,  the  available  literature  contains  a  shortage  of 
empirical  studies  relating  process  control  to  productivity 
or  product  costs.  To  date,  this  author  knows  of  only  five 
empirical  studies  which  considered  any  of  the  topics  covered 
above.  The  first  such  study  was  a  master's  thesis  by  Heff¬ 
ner  and  Weimer  (18)  that  sought  to  determine  which  factors 
motivated  government  subcontractors  to  undertake  capital 
equipment  investments.  Heffner  and  Weimer  found  the  most 
significant  factor  to  be  the  "opportunity  to  provide  a 
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better  quality  product  [18:82]. ”  However,  this  study  made 
no  attempt  to  correlate  this  finding  with  productivity  or 
cost  data. 

The  second  empirical  study  was  an  analysis  of  new  sheet 

» 

metal  forming  methods  by  Gossard,  et  al  (15).  This  study 
did  report  the  estimated  saving  associated  with  adaptive 
brakeforming  (15:45-46),  but  their  calculations  only  con¬ 
sidered  reductions  in  set-up  time  for  the  brakeforming  pro¬ 
cess.  While  such  measures  are  often  used  in  calculating 
productivity,  they  are  beyond  the  scope  of  this  study. 
Gossard,  et  al  did  report  other  potential  benefits,  includ¬ 
ing  improved  quality,  but  the  savings  associated  with  such 
benefits  were  not  tested. 

The  third  empirical  study  was  a  report  of  the  F-16  Tech 
Mod  integrated  quality  assurance  plan  by  Puma  (27).  He 
reported  (27:27-28,45-50,60-70)  significant  reductions  in 
reject  rates  for  rotational  machining  operations  and  numer¬ 
ical  control  punch  presses.  The  associated  reductions  in 
scrap  and  rework  were  of  the  magnitude  of  50-80Z,  and  Puma 
stated  this  translated  to  a  reduction  in  discrepancy  cost. 
The  reported  reductions  were  achieved  solely  through  im¬ 
proved  inspection  and  corrective  action  methods,  without 
significant  investment  in  new  capital  equipment.  This  lim¬ 
itation  has  been  considered  by  this  author,  and  the  Puma 
methods  will  be  evaluated  along  with  the  other  elements  of 
process  control  described  in  the  preceding  subsections. 


The  fourth  study  was  a  master's  thesis  by  Kruse  and 
Taras  (21)  that  evaluated  the  relationship  between  the  qual¬ 
ity  of  delivered  products  and  DoD  procurement  policies  and 
procedures.  Their  study  was  based  on  available  government 
data  regarding  defective  items  shipped  to  the  San  Antonio 
Air  Logistics  Center  (ALC),  and  shipments  made  under  the 
cognizance  of  Defense  Contracts  Administrative  Services 
Management  Area  (DCASMA)  Dayton.  One  finding  reported  by 
Kruse  and  Taras  was  that  contract  quality  performance  was 
independent  of  the  type  of  quality  inspection  clause  (e.g., 
M11-Q-9858A),  but  more  unacceptable  lots  occurred  when  the 
inspection  clauses  reflected  a  more  complex  product  (21:42). 
They  also  identified  other  factors  which  had  an  adverse 
effect  on  the  quality  of  spare  parts  delivered  to  the  San 
Antonio  ALC  (21:79)  —  such  as  competitive  market  environ¬ 
ment,  type  of  solicitation  and  relative  size  of  the  contrac¬ 
tor.  One  final  finding  reported  by  Kruse  and  Taras  was  that 
the  role  of  DCASMA  Dayton  regarding  defective  products  was 
reactionary  (21:80-81).  When  defective  products  were  repor¬ 
ted  by  the  using  activities,  DCASMA  Dayton  increased  the 
amount  of  government  corrective  action;  but,  when  the  DCASMA 
Dayton  quality  personnel  rejected  more  product  prior  to 
shipment,  the  number  of  defective  product  reports  from  the 
users  decreased.  In  general,  the  factors  affecting  product 
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quality  reported  by  Kruse  and  Taras  did  not  address  the 
elements  of  process  control  identified  above,  and  they  are 
thus  beyond  the  scope  of  this  study. 

The  final  empirical  report  was  a  master's  thesis  by 
Berry  and  Bland  (2)  that  proposed  a  model  to  measure  the 
relative  productivity  of  individual  DoD  contractors.  Their 
model  based  the  measures  of  productivity  on  past  performance 
in  terms  of  input  and  output  costs  (2:4-5,27).  While  their 
measures  employed  some  of  the  same  elements  of  productivity 
as  the  definition  provided  in  Chapter  I,  in  application. 
Berry  and  Bland's  measures  only  incorporated  one  of  the 
process  control  elements  addressed  earlier  in  this  litera¬ 
ture  review.  They  used  capital  equipment  costs  in  two 
separate  calculations  for  their  model  (2:41),  but  they  found 
that  simple  measures  of  productivity  could  not  be  accepted 
without  further  examination  of  economic  implications  (2:85). 
Berry  and  Bland  went  on  to  recommend  that  their  model  be 
used  in  conjunction  with  data  on  scrap  and  rework  costs  to 
determine  the  productivity  associated  with  Air  Force  Tech 
Mod  programs  (2:90).  However,  they  did  not  test  this  recom¬ 
mendation,  so  no  general  conclusion  can  be  drawn  from  this 
facet  of  their  study.  The  role  of  scrap  and  rework  as  a 
measure  of  productivity  will  be  evaluated  as  an  element  of 
process  control  in  the  methodology  of  this  study,  with 
resultant  empirical  data  reported  in  Chapter  IV  of  this 


Summary  of  the  Literature 

The  literature  iapliea  that  productivity  gain8  from 
improved  quality  assurance  are  not  only  possible,  but  in  at 
least  one  nation  --  Japan  —  have  proven  to  be  feasible. 
The  leaders  of  American  industry  seemed  to  agree  with  these 
views  and  appear  motivated  to  respond  to  the  challenge. 
Many  of  the  sources  cited  above  indicated  that  one  approach 
to  improving  productivity  would  be  to  develop  improved  pro¬ 
cess  controls  which  should  lead  to  reduced  product  costs. 

The  means  of  accomplishing  the  desired  improvements 
were  also  addressed.  The  literature  cited  recognized  the 
need  to  modernize  and  maintain  the  capital  equipment  as  a 
source  of  assuring  better  process  control.  While  modern 
capital  equipment  is  only  one  element  of  process  control, 
according  to  Heffner  and  Weimer,  contractors  appear  to  re¬ 
cognize  it's  role  in  improving  the  product  quality. 

A  relationship  between  effective  process  control  and 
reduced  costs  was  implied  in  the  literature  cited  above,  but 
only  Puma  reported  actual  data  to  support  the  relationship. 
However,  the  Puma  study  only  considered  a  limited  range  of 
control  elements.  As  a  result,  it  can  only  be  stated  that 
quality  improvements  generated  as  a  result  of  process  con¬ 
trols  may  increase  productivity.  The  effect  of  the  increased 
productivity  on  product  costs  or  profits  has  not  been  empir¬ 
ically  tested,  nor  has  it  been  demonstrated  that  reduced 
costs  were  passed  on  to  the  Department  of  Defense. 


Significance  of  the  Findings  to  DoD 

One  serious  shortcoming  of  the  literature  reviewed  to 
date  is  the  lack  of  empirical  evidence.  Few  of  the  sources 
reported  were  legitimate  research  studies.  The  controls 
used  to  test  the  hypotheses  were  not  described.  A  second 
shortcoming  of  the  literature  is  the  lack  of  data  to  demon¬ 
strate  that  cost  savings  were  passed  on  to  DoD. 

As  a  result  of  the  shortcomings  identified  in  the 
preceding  paragraph,  this  study  will  attempt  to  evaluate  the 
relationships  among  process  controls,  productivity  and  cost. 
As  described  in  the  methodology  outlined  in  the  next  chap¬ 
ter,  this  evaluation  will  be  accomplished  by  conducting 
interviews  with  government  personnel  stationed  at  aerospace 
plant  representative  offices. 

This  approach  will  provide  some  measure  of  the  aware¬ 
ness  of  DoD  personnel  regarding  the  role  of  contractor 
quality  controls  over  production  processes.  This  study  will 
also  demonstrate  whether  a  relationship  exists  between  those 
quality  measures  and  increased  productivity,  and  ultimately, 
whether  a  relationship  exists  between  such  controls  and 
reduced  weapon  system  costs. 


III.  Research  Methodology 


Overview 

The  literature  review,  contained  in  Chapter  II,  con¬ 
cluded  that  there  was  little  empirical  evidence  relating 
process  controls  to  productivity  and  cost.  This  shortcoming 
led  to  the  selection  of  the  methodology  described  below. 
This  methodology  will  provide  answers  to  the  following  re¬ 
search  questions: 

1.  Are  DoD  in-plant  personnel  aware  of  a  relationship 
between  quality  assurance  process  controls  and  produc¬ 
tivity? 

2.  If  such  a  relationship  exists,  are  associated  cost 
savings  passed  on  to  DoD  in  future  weapon  systems 
procurement? 

The  approach  selected  for  evaluating  these  research 
questions  is  a  detailed  survey  of  randomly  selected  aero¬ 
space  facilities  known  as  DoD  plant  representative  offices. 
The  next  section  of  this  chapter  describes  why  this  methodo¬ 
logy  was  chosen.  This  justification  section  is  followed  by 
a  third  section  containing  the  details  of  the  survey  metho¬ 
dology.  The  actual  survey  instrument  is  described  in  this 
third  section,  together  with  a  review  of  the  validity  and 
reliability  of  the  proposed  approach. 

This  chapter  continues  with  a  section  describing  the 
population  to  which  the  results  of  this  study  apply.  The 
sample  selection  approach  is  also  identified  in  this  sec¬ 
tion,  together  with  a  justification  of  the  representative- 
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ness  of  the  sample.  The  fifth  section  of  this  chapter 
describes  the  data  collection  plan,  and  includes  identifica¬ 
tion  and  classification  of  the  survey  variables,  evidence  of 
data  validity,  and  assumptions  and  limitations  of  the  plan. 
Finally,  this  chapter  concludes  with  a  section  describing 
the  planned  statistical  analysis  of  the  sample  data. 


Analysis  of  the  literature  reviewed  in  Chapter  II  indi¬ 
cated  that  no  solution  has  been  developed  to  satisfy  the 
research  questions  listed  in  the  preceding  section.  The 
conclusion  of  Chapter  II  identified  two  shortcomings  of  the 
available  literature.  These  shortcomings  were  the  lack  of 
empirical  data  relating  process  controls  to  productivity  and 
cost,  and  the  fact  that  no  evidence  was  presented  to  demon¬ 
strate  that  cost  savings  were  passed  on  to  DoD.  As  a 
result,  this  study  was  designed  to  gather  data  which  will 
allow  future  development  of  hypotheses  to  test  whether  pro¬ 
cess  controls  are  related  to  productivity  and  cost.  The 
data  was  collected  by  means  of  structured  personal  inter¬ 
views  developed  around  the  investigative  questions  described 
in  Chapter  I.  More  details  regarding  the  interview  instru¬ 
ment  are  provided  in  the  next  section. 

The  structured  interview  approach  was  selected  for  two 
reasons.  First,  the  structured  interview  is  considered  an 
excellent  data  collection  technique  (13:293-294).  Emory 
listed  several  advantages  of  this  method,  emphasizing: 
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The  greatest  value  of  this  method  is  the  depth  and 
detail  of  information  that  can  be  secured.  It  far 
exceeds,  in  volume  and  quality,  the  information  we  can 
usually  secure  from  telephone  and  mail  surveys. 

The  second  reason  for  selecting  the  personal  interview 
methodology  was  the  time  constraint  on  this  thesis  project. 
A  written  survey,  to  be  sent  to  personnel  within  DoD,  must 
be  approved  by  four  organizations,  up  to  and  including  the 
Air  Staff  at  the  Pentagon  (1:120).  This  approval  cycle  is  a 
time-consuming  process  and,  in  light  of  the  previously  iden¬ 
tified  benefits  of  the  proposed  method,  is  considered  un¬ 
necessary. 

The  proposed  method  is  not  without  its  disadvantages. 
Emory  (13:294)  noted,  "the  method  is  costly,  both  in  money 
and  time."  However,  the  time  issue  associated  with  this 
method  is  less  than  that  associated  with  written  surveys,  as 
this  survey  instrument  only  required  two  levels  of  local 
review  (1:119).  The  money  constraint  did  present  a  problem, 
since  the  proposed  respondents  were  located  throughout  the 
continental  United  States  (CONUS). 

The  most  significant  hurdles  associated  with  the  metho¬ 
dology  selected  were  the  costs  of  traveling  to  the  personal 
interviews,  and  the  development  of  an  accurate  measuring 
instrument.  The  first  issue  was  resolved  by  obtaining  ap¬ 
provals  in  the  manner  specified  in  the  APIT  Graduate  Manage¬ 
ment  Program  Handbook  (1).  The  second  issue  is  addressed  in 
later  sections  of  this  chapter. 
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Interview  Instrument 


Introduction.  The  complete  interview  questionnaire  is 
contained  in  appendix  A  to  this  report.  This  section  de¬ 
scribes  the  salient  features  of  the  interview  instrument  and 
the  rationale  for  the  approach  taken  in  the  interviews.  The 
variables  to  be  isolated  by  the  interview  questions  are  also 
described  in  this  section.  This  section  concludes  with  a 
review  of  the  literature  regarding  the  validity  and  relia¬ 
bility  of  the  selected  method. 

The  interview  instrument  is  arranged  to  evaluate  the 
four  investigative  questions  identified  in  Chapter  I.  Each 
investigative  question  is  evaluated  in  accordance  with  the 
criteria  described  in  the  next  four  subsections.  The  inves¬ 
tigative  questions  will  not  be  directly  posed  to  the  respon¬ 
dents;  rather,  an  overall  evaluation  will  be  based  on  the 
composite  response  to  the  individual  measurement  questions 
associated  with  the  investigative  questions.  Emory  (13:65- 
66)  indicated  such  a  hierarchy  is  necessary  to  satisfactor¬ 
ily  answer  research  questions  such  as  those  listed  above. 

The  technique  used  to  develop  the  specific  measurement 
questions  is  described  by  Emory  (13:238).  He  stated  that 
measurement  questions  should  generally  progress  from  the 
simple  to  the  complex,  and  from  general  items  to  more  speci¬ 
fic  items  of  interest.  Emory  further  stated. 

The  objectives  of  this  procedure  are  to  learn  the 
respondent's  frame  of  reference  and  to  extract  the  full 
range  of  desired  information  while  limiting  the  distor¬ 
tion  effect  of  earlier  questions  on  later  ones. 
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Emory's  guidance  was  instrumental  in  the  development  of 
the  sequence  of  the  measurement  questions  described  below. 
This  exhaustive  series  of  questions  was  intended  to  capture 
all  the  essential  data  to  answer  the  research  questions. 

Evaluation  of  Production  and  Manufacturing  Processes. 
The  first  group  of  measurement  questions  was  designed  to 
investigate  the  question, 

1.  How  do  DoD  in-plant  personnel  evaluate  contractor 
control  of  production  and  manufacturing  processes? 

The  investigation  was  limited  to  coverage  of  only  three 
areas  of  process  control:  general  issues,  machining  and  heat 
treating  operations.  This  limitation  in  scope  was  required 
because  of  the  complexity  inherent  in  DoD  weapon  systems, 
since  the  number  of  manufacturing  techniques  and  processes 
available  to  contractors  is  almost  unlimited. 

The  "general  issue"  measurement  questions  were  intended 
to  identify  the  respondents'  frames  of  reference  regarding 
process  control.  These  questions  initially  sought  to  deter¬ 
mine  the  type  of  contractual  quality  requirements  levied  on 
the  contractor,  and  the  DoD  guidance  for  evaluating  the 
contractor's  performance  against  these  requirements.  This 
series  of  questions  then  became  more  specific,  so  that  the 


elements  of  process  control  which  may  play  a  role  in  the 
government  surveillance  were  identified.  The  "general  is¬ 
sue"  questions  concluded  by  ascertaining  the  respondents' 
definitions  of  process  control  and  yield  for  later  develop¬ 
ment  of  operational  definitions  based  on  this  study. 


The  individual  general  issue  questions  were  based 
primarily  on  the  elements  of  process  control  identified  in 
the  literature  review  in  Chapter  II.  In  addition  to  those 
sources  cited  in  Chapter  II,  H50,  Evaluation  of  a  Contrac¬ 


tor's  Qui 


Program  (34),  a  government  document  used  in 


conjunction  with  MIL-Q-9858A,  provided  the  guidance  for  the 
question  regarding  evaluation  of  en viroaaea tal  controls. 
The  objective  of  these  "general  issue"  questions  was  to 
elicit  data  that  has  broad  application  as  compared  to  the 
measurement  questions  asked  about  the  machining  and  heat 
treating  operations. 

The  "machining  operation"  questions  were  developed  to 
measure  the  level  of  process  control  surveillance  performed 
by  the  government.  The  questions  tended  to  be  much  more 
specific  than  those  listed  in  the  "general  issues"  section. 
The  sequence  of  questions  was  not  as  important  as  above, 
since  specificity  regarding  machining  process  controls  was 
the  objective  of  this  group  of  measurement  questions.  How¬ 
ever,  an  attempt  was  made  to  present  these  questions  in  an 
operational  sequence,  proceeding  from  equipment  maintenance 
through  actual  machine  operation  to  component  inspection. 
Finally,  the  "machining"  measurement  questions  ended  with  an 
evaluation  of  how  the  government  personnel  evaluate  advanced 
machining  process  controls. 
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The  controls  evaluated  in  the  "machining”  section  of 
the  interview  were  also  based  on  those  process  elements 
cited  in  the  literature  review.  In  addition  to  the  previous 
sources,  H50  (34:17,25-26)  provided  reference  material  to 
develop  the  questions  regarding  production  tooling  used  as  a 
media  of  inspection  and  equipment  set-up  and  operation. 
Bohlen  and  Sweeney  (3:28-31,65-68,71-72)  presented  informa¬ 
tion  that  was  used  to  develop  the  question  on  statistical 
process  control,  and  Puma  (27:69)  provided  additional  data 
on  some  of  the  elements  of  advanced  machining  techniques. 

The  measurement  questions  used  to  evaluate  the  level  of 

government  surveillance  of  "heat  treating  operations"  con- 
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eluded  the  first  section  of  the  interview.  Like  the  preced¬ 
ing  section  on  "machining  operations,"  these  questions  were 
organized  in  a  sequential  format,  proceeding  from  material 
preparation  through  inspection  of  the  processed  components. 
As  was  true  for  the  "machining"  questions,  the  objective 
here  was  to  understand  how  government  surveillance  is  accom¬ 
plished  so  that  conclusions  could  be  drawn  regarding  the 
evaluation  of  production  and  manufacturing  processes. 

The  "heat  treating"  questions  presented  here  were  based 
on  the  general  controls  identified  in  two  common  military 
specifications  dealing  with  heat  treating  of  aluminum  and 
steel  (35;  36).  Regardless  of  the  source  material,  the 
questions  were  designed  to  measure  basic  understanding  of 
the  features  of  the  process  in  lieu  of  the  detailed  require- 
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nents  of  a  given  specification.  This  perspective  helped  to 
assure  the  data  developed  vere  related  to  the  investigative 
question  listed  above. 

Evaluation  of  the  Contractor  Cost  Controls.  This  sub¬ 
section  describes  the  second  group  of  measurement  questions, 
which  were  designed  to  investigate  the  question, 

2.  How  do  DoD  in-plant  personnel  evaluate  the  contrac¬ 
tor's  cost  controls? 

This  series  of  questions  began  by  establishing  the 
contractual  framework  and  definition  for  control  of  noncon¬ 
forming  material.  This  step  was  necessary  because  of  the 
correlation  that  is  made  in  the  government  specifications 
between  nonconforming  material  and  cost  control  (11:5,8,12; 
39:3).  These  measurement  questions  became  more  specific  as 
they#  progressed  through  the  elements  of  nonconforming  mater¬ 
ial  to  the  elements  of  cost  control. 

The  criteria  reviewed  in  this  subsection  were  based  on 
controls  typically  required  by  MIL-Q-9858A  and  MIL-STD- 
1520A,  as  identified  in  the  literature  review  of  this  re¬ 
port.  In  addition  to  the  cited  documents,  the  question 
regarding  the  possible  savings  associated  with  reduced  sam¬ 
ple  sizes  was  drawn  from  Bohlen  and  Sweeney  (3:47,61)  and 
Puma  (27:44,72).  The  final  question  in  this  subsection  was 
based  on  the  discrepancy  cost  criteria  contained  in  H50 
(34:13),  the  government  guide  to  evaluating  a  contractor's 
MIL-Q-9858A  program. 


This  group  of  measurement  questions  provided  data  and 
background  for  proceeding  to  the  next  level  of  specificity 
in  the  overall  interview  design.  Having  established  a 
framework  for  what  constitutes  government  evaluation  of 
process  and  cost  controls,  the  Investigation  proceeded  to  an 
evaluation  of  the  relationships  among  those  two  control 
elements  and  productivity. 

Relationships  Among  Process  Controls.  Cost  and  Produc¬ 
tivity.  This  third  group  of  measurement  questions  was  de¬ 
signed  to  investigate  the  question, 

3.  If  the  DoD  in-plant  personnel  evaluate  the  process 
controls,  how  do  they  define  a  relationship  between 
such  controls  and  productivity  improvement  or  cost? 

This  investigation  was  more  specific  in  the  overall  instru¬ 
ment  design  than  those  in  the  previous  two  subsections.  Like 
the  preceding  "subsections,  the  internal  sequence  within  this 
group  of  measurement  questions  flowed  from  the  general  to 
the  specific  items  of  interest. 

Like  the  previous  investigative  sequences,  this  subsec¬ 
tion  began  with  questions  designed  to  establish  the  frame¬ 
work  for  the  measurement  questions  that  follow.  An  evalua¬ 
tion  of  the  elements  of  productivity  improvement,  as  de¬ 
scribed  in  the  literature  review,  then  followed.  The  final 
questions  of  this  subsection  sought  to  determine  the  rela¬ 
tionships  among  productivity  and  the  process  and  cost  con¬ 
trols  evaluated  in  the  preceding  investigative  sequences. 
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Successful  completion  of  this  set  of  measurement  ques¬ 
tions  permitted  an  evaluation  of  the  relationships  among 
process  controls,  productivity  and  cost.  While  this  would 
satisfy  the  original  research  question,  it  would  not  address 
the  second  shortcoming  noted  in  the  present  literature. 
This  shortcoming  —  the  lack  of  evidence  that  cost  savings 
are  passed  on  to  DoD  --  is  addressed  in  the  final  set  of 
measurement  questions  described  below. 

Evaluation  of  Supporting  Data.  This  group  of  measure¬ 
ment  questions  was  designed  to  evaluate  the  final  investiga¬ 
tive  question, 

4.  What  data  is  available  to  demonstrate  relationships 
among  process  controls,  productivity  improvement  and 
reduced  costs? 

These  questions  represent  the  final  level  of  specificity  in 
this  interview.  As  before,  the  sequence  began  with  ques¬ 
tions  designed  to  establish  the  framework  for  government 
analysis  of  contractor  cost  data.  The  sequence  continued 
with  an  evaluation  of  the  way  government  personnel  assess 
the  reliability  of  the  contractor  data  identified  in  the 
previous  subsection.  The  final  two  questions  sought  to 
determine  whether  the  available  data  is  used  to  assure 


reduced  costs  for  the  government. 

Analysis  of  the  data  from  the  preceding  four  investiga¬ 
tive  questions  will  provide  information  to  overcome  the 
shortcomings  of  the  current  literature,  as  described  in  the 
introduction  to  this  section.  This  report  will  also  provide 


empirical  evidence  to  describe  the  relationships  among  pro¬ 
cess  controls,  productivity  and  cost.  And,  the  equally 
important  objective  of  determining  whether  the  relationships 
among  these  elements  results  in  reduced  weapon  system  costs 
will  be  accomplished. 

Personal  Data.  In  addition  to  the  preceding  investiga¬ 
tive  questions,  the  interview  instrument  contained  five 
questions  to  identify  the  survey  respondent.  The  respon¬ 
dents  will  remain  anonymous;  however,  this  data  was  collec¬ 
ted  to  assure  the  respondent  can  be  contacted  if  necessary 
for  clarification  or  further  discussions.  Emory  (13:238) 
recommended  that  personal  questions  be  included  at  the  end 
of  the  interview  to  reduce  the  threat  or  demotivation  such 
questions  might  cause  if  asked  too  soon.  He  noted,  "Ques¬ 
tions  which  are  more  interesting,  easier  to  answer,  and  less 
threatening  usually  are  placed  early  in  the  sequence  to 
encourage  response  and  promote  rapport  [13:222]." 

Validity  and  Reliability  of  the  Interview  Instrument. 
In  discussing  the  validity  of  personal  interviews,  Emory 
(13:214)  stated, 

A  major  criticism  of  the  survey  method  is  that  it 
depends  so  completely  on  verbal  behavior.  The  respon¬ 
dent  can  knowingly  give  untrue  or  misleading  answers. 

But,  Emory  also  recognized  that  questionnaires  such  as  the 

one  described  above  are  appropriate  "when  the  objective  is 

to  discover  opinions  and  degrees  of  knowledge  [13:234]."  In 

consideration  of  these  factors,  Emory  believed  it  was  incum- 
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bent  upon  the  researcher  to  assure  the  survey  instrument 
selected  was  an  accurate  measurement  tool,  characterized  by 
its  validity  and  reliability  (13:128). 

To  be  considered  a  valid  measuring  tool,  the  survey 
instrument  must  provide  adequate  coverage  of  the  topic  under 
consideration.  Emory  noted,  "If  the  instrument  contains  a 
representative  sample  of  the  universe  of  subject  matter  of 
interest,  then  content  validity  is  good  [13:129]. n  Emory 
further  suggested  that  the  best  method  for  assuring  the  con¬ 
tent  validity  is  by  having  the  instrument  reviewed  by  a 
panel  of  experts  (13:129).  This  type  of  review  will  elimi¬ 
nate  the  bias  caused  by  inclusions  or  omissions  in  the 
survey,  which  Emory  identified  as  the  greatest  source  of 
distortion  in  surveys  (13:226,229).  The  recommended  instru¬ 
ment  review  was  accomplished  on  this  survey  instrument, 
using  local  experts  from  the  Air  Force  Institute  of  Techno¬ 
logy  faculty.  These  experts  are  identified  in  appendix  C  to 
this  report,  along  with  a  short  synopsis  of  their  experience 
in  the  areas  covered  by  the  survey  instrument. 

The  second  characteristic  of  a  good  measurement  tool  is 
reliability.  Emory  (13:132)  stated,  "A  measure  is  'reli¬ 
able'  to  the  degree  that  it  supplies  consistent  results." 
While  this  characteristic  cannot  be  determined  without  some 
type  of  preliminary  testing,  Emory  suggested  that  reliabil¬ 
ity  can  be  improved  by  assuring  the  external  sources  of 
variation  are  minimized  (13:134).  In  a  structured  inter- 


view,  the  sources  of  variation  are  reduced  by  asking  ques¬ 
tions  in  a  standardized  format  and  sequence  (13:215).  The 
time  allotted  and  the  money  available  for  this  study  did  not 
permit  pretesting  of  the  interview  instrument.  As  a  result, 
the  reliability  of  this  interview  instrument  was  based  sole¬ 
ly  on  the  standardized  format  and  sequence  of  the  questions 
asked  of  the  respondents. 

Population  and  Sample  Description 

The  initial  population  under  consideration  was  aero¬ 
space  plant  representative  offices  under  the  cognizance  of 
various  DoD  procurement  offices.  These  facilities  are  often 
identified  by  their  acronyms  as  AFPROs,  ARPROs,  DCASPROs  and 
NAVPROs,  and  number  approximately  80  within  the  continental 
U.S.  However,  when  the  scope  of  this  study  was  narrowed  to 
facilities  performing  in-house  machining  and  heat  treating 
operations,  the  population  became  further  restricted.  To 
determine  the  exact  population  of  aerospace  facilities  per¬ 
forming  these  specific  processes,  telephonic  contact  was 
made  with  all  plant  representative  offices  listed  in  DoD 
4105. 59-H,  the  DoD  Directory  of  Contract  Administration 
Services  Components  (33:p.I-l  to  1-17).  This  preliminary 
contact  resulted  in  a  study  population  of  43  facilities, 
each  of  which  contained  all  of  the  relevant  dimensions 
described  below. 
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The  most  relevant  dimensions  of  this  population  were 
the  criteria  that  the  facility  must  be  an  aerospace  plant 
representative  office  listed  in  DoD  4105. 59-H,  currently 
performing  the  machining  and  heat  treating  operations  listed 
in  the  interview  instrument.  Another  important  dimension  of 
the  population  was  the  requirement  that  the  potential  re¬ 
spondents  must  be  responsible  for  administering  the  floor- 
level  quality  assurance  program  within  the  plant  representa¬ 
tive  office  and  have  knowledge  of  the  use  and  maintenance  of 
the  contractor  quality  cost  controls.  This  limitation  as¬ 
sured  selection  of  personnel  with  the  most  current  knowledge 
of  the  contractors'  production  and  manufacturing  process  and 
cost  controls. 

The  preceding  limitations  resulted  in  a  population 
characterized  by  facilities  primarily  involved  in  the  manu¬ 
facture  of  mechanical  and  structural  aerospace  systems  and 
subsystems.  Typical  items  manufactured  in  such  facilities 
include  airframes,  propulsion  systems,  actuator  systems, 
etc.  Excluded  from  the  population  were  those  facilities 


two-digit  number.  The  sample  facilities  were  identified 
using  the  last  two  digits  of  a  random  number  table.  A 
random  number  table  contained  in  a  college-level  statistics 
book  was  considered  adequate  for  this  purpose.  Alternate 
sample  facilities  were  also  identified  in  case  permission 
could  not  be  obtained  to  perform  the  on-site  survey,  or  some 
later  indication  was  received  that  the  facility  did  not  meet 
the  relevant  dimensions  outlined  above.' 

The  sampling  plan  described  above  helped  assure  the 
data  generated  as  a  result  of  this  study  could  be  general¬ 
ized  to  the  Identified  population.  Appendix  B  of  this 
report  list  all  the  facilities  that  met  the  population 
dimensions  described  above.  The  next  two  sections  of  this 
chapter  provide  additional  detail  on  the  data  collection 
plan  and  the  statistical  tests  used  to  ensure  the  adequacy 
of  the  data  developed  in  this  research  study.  These  details 
must  also  be  considered  when  evaluating  whether  the  results 
can  be  generalized  to  the  population  under  consideration. 

Data  Collection  Plan 

This  section  contains  the  details  of  how  the  data  were 
collected,  as  well  as  the  scoring  criteria,  variable  identi¬ 
fication  and  classification,  and  evidence  of  the  data  valid¬ 
ity.  These  details  are  outlined  in  the  subsections  below, 
and  are  followed  by  a  final  subsection  describing  the  as¬ 
sumptions  and  limitations  associated  with  this  method. 
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Specifics  of  the  Plan.  As  noted  in  the  preceding 
sections,  the  interviews  to  collect  the  data  were  conducted 
at  five  randomly  selected  aerospace  facilities  identified  as 
government  plant  representative  offices.  As  described  by 
Emory,  "What  is  called  a  ' survey '  in  common  parlance  is  an 
ex  post  facto  study  (without  manipulation  of  variables)  in 
the  interrogation  format  [13:87]."  Emory  also  noted  that  an 
ex  post  facto  design  allows  the  researcher  to  only  report 
what  has  happened,  without  any  attempt  to  influence  the 
variables,  because  manipulation  of  the  variables  could  in¬ 
troduce  bias.  This  caution  does  not  preclude  manipulation 
of  the  resultant  data,  as  described  in  the  next  section. 

As  an  ex  post  facto  study,  this  report  provides  de¬ 
scriptive  data  to  answer  the  research  questions  identified 
above.  As  noted  in  Chapter  I,  the  typical  objectives  of 
such  a  study  are: 

(1)  descriptions  of  phenomena  or  characteristics  asso¬ 
ciated  with  a  subject  population,  (2)  estimates  of  the 
proportions  of  a  population  that  have  these  character¬ 
istics,  [and]  (3)  discovery  of  associations  among  dif¬ 
ferent  variables  [13:91]. 

Emory  described  a  number  of  methods  to  assure  these  objec¬ 
tives  were  met.  Emory  also  recognized  (13:109)  that  the 
best  research  technique  to  neutralize  the  confounding  af¬ 
fects  of  other  variables  —  random  assignment  of  variables 
in  some  form  of  test  treatment  —  was  unavailable  in  the  ex 
post  facto  design.  But  he  noted  that  extraneous  variation 
could  be  reduced  by  selecting  subjects  who  possess  certain 


control  characteristics.  The  control  recommended  bj  Emory 
was  achieved  at  each  plant  representative  office  by  inter¬ 
viewing  respondents  considered  to  be  the  most  knowledgeable 
of  the  process  and  cost  controls  under  investigation.  This 
criterion  was  met  through  an  introductory  letter  requesting 
that  such  experts  be  made  available,  including  Quality  As¬ 
surance  (QA)  supervisors,  QA  floor-level  surveillance  spec¬ 
ialists,  and  cost/price  analysts  as  appropriate. 

In  addition  to  controlling  confounding  or  extraneous 
variables,  the  descriptive  study  must  also  meet  the  chal¬ 
lenge  of  obtaining  complete  and  accurate  information.  One 
of  the  best  methods  to  accomplish  this  objective  is  through 
motivation  of  the  respondent  (13:228).  Emory  noted  that 
motivation  can  be  achieved  by  establishing  good  rapport  with 
the  respondent,  and  providing  assurances  that  answers  will 
be  kept  confidential  and  used  only  in  combined  statistical 
totals.  This  study  attempted  to  accomplish  respondent  moti¬ 
vation  by  providing  the  recommended  assurances  at  the  begin¬ 
ning  of  each  interview.  Further  efforts  to  establish  rap¬ 
port  were  accomplished  by  providing  background  information 
on  the  purpose  and  general  focus  of  the  research,  as  well  as 
background  information  on  the  interviewer.  Finally,  the 
respondents  were  requested  to  provide  answers  which  reflect 
a  policy  viewpoint  regarding  government  surveillance,  as 
opposed  to  personal  perspective. 


Scoring.  Grouping  and  Summarizing  the  Data.  The  first 
step  to  provide  meaning  to  the  data  collected  is  the  devel¬ 
opment  of  a  scale.  Emory  described  some  of  the  methods  used 
to  develop  scales  (13:258-260),  including  the  consensus 
scale,  in  which  a  panel  of  judges  evaluate  the  items  chosen 
to  determine  whether  they  are  relevant  to  the  topic  under 
consideration  and  unambiguous  in  implementation.  He  further 
stated  that  this  approach  can  be  used  to  determine  the 
validity  of  the  scale,  especially  when  scaling  abstract 
concepts.  Since  this  study  is  examining  abstract  relation¬ 
ships,  evaluation  of  the  measurement  instrument  was  accom¬ 
plished  by  a  group  of  professors  (see  appendix  C)  at  the  Air 
Force  Institute  of  Technology  (AFIT).  Their  recommendations 
regarding  the  relevance  and  clarity  of  the  instrument  were 
incorporated  prior  to  the  actual  interviews. 

As  noted  in  a  previous  section,  the  final  scale  will  be 
based  on  the  overall  responses  associated  with  each  investi¬ 
gative  question.  This  technique  will  result  in  a  scale 
where  the  data  from  the  individual  investigative  questions 
are  used  to  determine  whether  relationships  exist  among 
process  control,  productivity  and  weapon  system  costs.  Data 
collected  and  organized  in  such  a  fashion  is  known  as  nomi¬ 
nal  or  classif icatory  data.  Development  of  operational 
definitions  to  achieve  the  appropriate  classes  is  described 
in  the  next  subsection. 


The  purpose  of  classification  is  to  facilitate  analysis 
of  the  data.  As  noted  by  Emory  (13:369),  "Analysis  involves 
breaking  down  and  rearranging  data  into  more  meaningful 
groups  to  aid  the  search  for  significant  relationships." 
Emory  called  this  process  coding  (13:371),  and  he  recognized 
that  such  coding  sacrifices  some  of  the  detail  but  is  neces¬ 
sary  for  efficient  analysis.  Emory  described  four  tests 
(13:371-372)  to  determine  whether  the  categories  selected 
were  adequate.  This  study  met  the  first  test  --  the  cate¬ 
gories  are  appropriate  for  the  research  problem  and  purpose 
—  when  the  previously  mentioned  faculty  panel  concurred 
that  the  research  instrument  provided  information  to  answer 
the  investigative  questions.  The  next  two  tests  require  the 
categories  to  be  exhaustive  and  mutually  exclusive.  The 
faculty  panel  also  evaluated  the  interview  instrument  for 
these  criteria,  to  assure  there  was  a  category  for  each  data 
item  and  that  each  data  item  fell  into  only  one  category. 
The  final  test  —  the  categories  are  derived  from  one  clas¬ 
sification  principle  —  were  met  when  the  panel  determined 
every  class  in  the  category  set  was  defined  in  terms  of  one 
concept . 

Identification  and  Classification  o f  Variables.  The 
variables  to  be  evaluated  by  this  study  will  be  based  on  the 
operational  definitions  developed  for  the  four  categories 
described  below.  As  stated  by  Emory  (13:372),  one  of  the 


functions  of  such  definitions  is  to  provide  categories  com- 


posed  of  mutually  exclusive  elements.  Emory  further  stated 
that  operational  definitions  nshould  specify  the  character¬ 
istics  to  be  observed  and  how  they  are  to  be  observed 
[13:28]."  The  purposes  cited  by  Emory  will  be  accomplished 
in  Chapter  IV,  when  the  actual  operational  definitions  are 
derived  from  the  data  in  the  respondents'  answers.  Since 
this  is  a  descriptive  study,  this  approach  will  assure  the 
current  study  does  not  prejudice  the  data,  and  will  permit 
development  of  testable  variables  for  follow-on  studies. 

The  four  operational  definitions  to  be  developed  in 
Chapter  IV  are:  (1)  adequate  process  controls;  (2)  adequate 
cost  controls;  (3)  improved  productivity;  and  (4)  associated 
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cost  reductions.  Each  variable  will  be  based  on  the  cumula¬ 
tive  responses  to  the  four  investigative  questions  described 
earlier  in  this  chapter.  The  following  subsections  describe 
the  validity  measures  that  apply  to  these  variables,  and  the 
assumptions  and  limitations  of  this  methodology. 

Evidence  of  Data  Validity.  General  considerations 
regarding  the  validity  and  reliability  of  the  interview 
instrument  were  described  in  the  third  section  of  this 
chapter.  In  addition,  the  validity  of  the  classification 
scheme  was  described  in  the  subsection  on  scoring,  grouping 
and  summarizing  the  data.  This  subsection  is  concerned  with 
the  validity  of  the  data  obtained  as  a  result  of  the  overall 
methodology.  Emory  (13:294)  provided  the  following  criteria 
to  measure  the  effectiveness  of  personal  interviews: 


Three  broad  conditions  are  needed  for  a  successful 
personal  interview.  They  are  (1)  accessibility  of  the 
needed  information  to  the  respondents,  (2)  understand¬ 
ing  of  the  respondents  of  their  roles,  and  (3)  motiva¬ 
tion  of  the  respondents  to  accept  such  a  role  and 
fulfill  its  requirements. 

These  criteria  were  met  by  the  methodology  described  above. 
As  previously  noted,  the  respondents  selected  were  those 
having  direct  knowledge  of  the  floor-level  controls  and  cost 
data  at  each  facility.  Consideration  was  also  given  to  the 
respondents'  understanding  and  motivation  during  the  inter¬ 
view  orientations  at  each  facility. 

Two  additional  tests  of  data  validity  were  performed  by 
this  researcher  at  each  facility.  Where  supporting  evidence 
was  claimed,  the  researcher  verified  the  data  was  available 
and  actually  represented  the  respondents'  statements.  In 
the  second  test,  the  researcher  personally  reviewed  the 
checklists  used  by  the  government  personnel  to  evaluate  the 
contractors'  process  and  cost  controls.  As  described  in  the 
"Vita”  included  with  this  report,  this  researcher's  back¬ 
ground  provides  the  necessary  expertise  to  evaluate  check¬ 
lists  developed  to  assess  the  adequacy  of  quality  assurance 
controls  implemented  in  accordance  with  MIL-Q-9858A  and  MIL- 
STD-1520A. 

Assumptions  and  Limitations.  As  noted  in  the  section 
justifying  the  methodology  selected,  there  are  both  time  and 
money  constraints  which  limited  the  depth  of  this  research 
project.  In  addition  to  the  previously  identified  time 


constraint,  it  must  be  recognized  that  the  selected  methodo- 


logy  was  further  limited  by  the  time  available  for  on-site 
interviews.  The  facilities  meeting  the  population  dimen¬ 
sions  described  above  were  located  throughout  the  continen¬ 
tal  U.S.  Because  of  commitments  to  attend  regularly  sche¬ 
duled  classes  in  the  researcher's  graduate  program,  there 
was  a  limited  period  of  time  between  academic  quarters  in 
which  to  accomplish  the  interviews  at  these  remote  facili¬ 
ties.  Together  with  the  previously  identified  cost  con¬ 
straint,  this  final  time  constraint  dominated  the  decision 
to  limit  the  sample  size  for  this  study. 

According  to  Emory  (13:148-149,152,160-164,299-301), 
sample  size  is  but  one  of  three  considerations  the  research¬ 
er  must  make  to  reduce  bias  in  personal  interview  situa¬ 
tions.  The  other  two  considerations  --  nonresponse  and 
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response  error  —  are  discussed  below.  In  discussing  sam¬ 
pling  error,  Emory  noted  that  the  attribute  measures  used 
with  nominal  data  are  the  variance  and  standard  error  (or 
standard  deviation).  He  provided  a  formula  to  determine  an 
appropriate  sample  size  based  on  subjective  decisions  re¬ 
garding  the  interval  size  and  degree  of  confidence  desired, 
corrected  for  a  finite  population.  This  formula  is  listed 
below  as  equation  (1): 

std  error  -  [ pq/(n-l ) ] 1/2 [ (N-n) / ( N-l ) ] 1 /2  (1) 

where:  p  ■  proportion  of  population  having  an  attribute 

q  ■  proportion  of  population  not  having  an  attribute 
N  -  size  of  population 
n  m  size  of  sample 


Emory  noted  that  it  is  possible  that  the  researcher  has 
no  information  regarding  the  probable  values  for  j>  and  in 
such  cases,  the  researcher  can  assume  j>  -  0.5.  In  this 
study,  this  assumption  was  made  in  order  to  evaluate  the 
most  appropriate  sample  size.  In  addition  to  assuming  the 
value  of  j>,  a  decision  was  made  to  assume  a  sample  interval 
of  +/-  0.1  around  the  assumed  2  value  and  a  confidence  level 
of  95%.  Using  these  values  in  equation  (1),  an  appropriate 
sample  size  can  be  calculated  by  solving  the  equation  for  n. 
The  basic  equation  is: 

0.1/1.96  «  [(0.5)(0.5)/(n-l)jl/2[(43-n)/(43-l)]l/2 

The  value  1.96  in  this  equation  is  associated  with  the 
95%  confidence  level  (13:163).  The  solution  of  this  equa¬ 
tion  indicated  the  appropriate  sample  size  would  be  30.23 
(rounded  to  31)  to  obtain  the  desired  interval  and  confi¬ 
dence  level.  Obviously,  this  sample  size  was  impractical  in 
light  of  the  time  and  money  constraints,  so  it  was  assumed 
that  the  sampling  error  could  not  be  controlled  in  this 
study.  Even  with  a  wider  interval  (  +  /-  0.2)  and  a  reduced 
confidence  Interval  (90%),  the  required  sample  size  would 
have  been  prohibitive.  As  a  result,  the  operational  defini¬ 
tions  developed  in  Chapter  IV  will  prescribe  the  acceptable 
levels  of  accuracy  for  this  study.  The  actual  standard 
error  and  confidence  level  will  also  be  calculated  in  chap¬ 
ter  IV  based  on  data  gathered  during  the  on-site  surveys. 
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The  second  source  of  bias  identified  by  Emory  (13:299) 
—  nonresponse  error  —  results  when  a  respondent  selected 
for  the  sample  cannot  be  located.  This  source  was  not 
considered  a  problem  for  this  study,  as  the  preliminary 
contact  with  the  various  plant  representative  offices  indi¬ 
cated  a  high  degree  of  acceptance  for  a  planned  visit.  The 
third  source  of  bias  in  personal  interview  situations  — 
response  error  —  occurs  when  a  respondent  reports  differ¬ 
ently  than  the  actual  facts  (13:301).  As  noted  in  the 
preceding  subsection,  a  number  of  tests  were  used  during  the 
interviews  to  assure  the  validity  of  the  data  reported  by 
the  respondents. 

Two  final  assumptions  must  be  made  about  the  proposed 
methodology.  First,  the  independence  of  the  sample  facili¬ 
ties  visited  is  generally  considered  necessary  in  order  to 
generate  valid  results  (13:413).  This  requirement  was  as¬ 
sumed  to  be  met  because  the  various  facilities  are  widely 
dispersed  and  each  is  headed  by  a  single  military  Commander. 
Second,  since  the  various  test  prescribed  for  the  interview 
instrument  were  met,  the  data  obtained  can  be  generalized  to 
the  entire  population.  This  generalization  applies  to  plant 
representative  office  known  as  ARPROs  and  NAVPROs,  even 
though  none  were  selected  for  evaluation  as  a  result  of  the 
small  sample  size  and  the  random  sampling  techniques. 


Statistical  Tests 

This  final  section  briefly  describes  the  statistical 
tests  which  are  normally  applied  to  personal  interview  data. 
However,  because  of  the  limitations  of  the  proposed  sample 
size  and  the  associated  risk  of  sampling  error,  these  tests 
will  not  be  applied  to  this  data  set.  Instead,  this  re¬ 
searcher  will  report  the  exact  number  and  kind  of  responses 
to  each  investigative  question.  The  results  will  then  be 
further  reduced  by  comparing  the  cumulative  responses  to  the 
operational  definitions  developed  in  Chapter  IV.  The  rea¬ 
sons  for  reporting  exact  values  were  described  by  Emory 

(13:380),  when  he  noted  that  percentages  hide  the  base  from 
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which  they  have  been  computed,  especially  when  categories 
have  been  made  up  of  small  numbers  as  are  inherent  in  this 
methodology. 

Emory  (13:388-390)  described  a  method  to  analyze  nomi¬ 
nal  data  for  correlation.  In  his  methodology,  he  required 
classification  of  each  variable  into  two  or  more  categories, 
followed  by  a  cross-classification  of  each  variable  into  a 
two-way  table.  This  technique  facilitates  analysis  of  the 
data,  and  also  prepares  the  data  for  later  statistical 
tests.  This  classification  process  will  be  accomplished  in 
Chapter  IV  to  portray  any  relationships  that  might  exist, 
but  no  further  statistical  analysis  will  be  performed  for 
the  reasons  described  above. 
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In  tern  of  actual  atatistlcal  tests,  Emory  (13:405- 
408,413-415,434)  described  the  nonparametric  tests  normally 
applied  to  oaa-aaaple  nominal  data.  His  description  in¬ 
cluded  guidance  about  assuring  statistical  significance, 
developing  the  null  and  alternative  hypotheses,  establishing 
decision  rules  for  accepting  or  rejecting  the  null  hypothe¬ 
sis,  and  using  the  binomial  and  C  tests.  In  the  case  of 
this  study,  Emory's  guidance  is  not  applicable,  but  it  is 
recommended  that  readers  review  his  guidance  if  they  plan  to 
replicate  this  study  or  attempt  any  of  the  future  studies 
proposed  in  Chapter  V  of  this  report. 


Overview 


Chapter  II  of  this  report  contained  a  review  of  the 
literature  relating  process  controls  to  productivity  and 
cost.  It  was  concluded  there  was  little  empirical  evidence 
at  this  time  to  relate  such  variables  to  each  other.  This 
shortcoming  in  the  current  literature  led  to  the  selection 
of  the  survey  methodology  described  in  Chapter  III,  which 
was  developed  to  answer  the  following  research  questions: 

1.  Are  DoD  in-plant  personnel  aware  of  a  relationship 
between  quality  assurance  process  controls  and  produc¬ 
tivity? 

2.  If  such  a  relationship  exists,  are  associated  cost 
savings  passed  on  to  DoD  in  future  weapon  system  pro¬ 
curements? 

This  chapter  reports  the  findings  generated  by  the 
survey  described  in  Chapter  III,  and  provides  an  in-depth 
analysis  of  the  results.  The  body  of  this  chapter  begins 
with  a  section  describing  the  implementation  of  the  survey 
and  the  deviations  from  the  planned  design.  This  descrip¬ 
tion  includes  a  discussion  of  the  validity  and  reliability 
measures  actually  accomplished  during  the  performance  of  the 
survey . 

This  description  section  is  followed  by  a  third  section 
reporting  the  actual  results  of  the  survey  and  an  analysis 
of  those  results.  The  results  are  initially  grouped  to 
provide  answers  to  each  of  the  four  investigative  questions 


described  in  Chapter  III.  The  third  section  continues  with 
an  analysis  of  the  findings  in  terms  of  the  investigative 
questions  and  the  original  research  questions.  This  analy¬ 
sis  requires  a  series  of  data  reductions  to  develop  the 
operational  definitions  described  in  Chapter  III.  This 
chapter  then  concludes  with  a  fourth  section  containing 
statistical  tests  appropriate  for  the  methodology  developed. 
These  tests  will  determine  whether  the  data  developed  by 
this  study  can  be  generalized  to  the  population  of  plant 
representative  offices  identified  in  Chapter  III. 


Description  of  the  Survey 

Chapter  III  described  the  survey  design  planned  for 
this  study.  This  section  describes  how  the  survey  was 
actually  accomplished  and  explains  the  deviations  from  the 
design.  This  description  also  includes  a  discussion  of  the 
validity  and  reliability  measures  actually  performed  during 
the  on-site  visits  to  the  plant  representative  offices. 

The  planned  approach  called  for  an  in-depth  survey  of 
five  randomly  selected  aerospace  plant  representative  of¬ 
fices  under  the  cognizance  of  various  DoD  procurement  activ¬ 
ities.  The  universe  of  potential  facilities  included  43 
plant  representative  offices  listed  in  DoD  4105. 59-H,  as 
identified  in  appendix  A,  that  were  currently  performing  the 
machining  and  heat  treating  operations  addressed  in  the 
interview  instrument.  A  deviation  from  this  plan  occurred 
when  AFPRO  General  Dynamics,  Fort  Worth  Division,  refused 


this  researcher's  request  for  permission  to  perform  an  on¬ 
site  surrey.  Their  rationale  vas  they  had  just  undergone  a 
similar  study,  and  the  available  data  could  be  obtained  from 
the  Air  Force  Contract  Management  Division  at  Kirtland  AFB. 
However,  as  noted  in  Chapter  II  of  this  report,  this  author 
was  unaware  of  any  empirical  research  designed  to  test  the 
relationships  described  by  the  present  study. 

The  impact  of  the  deviation  from  the  randomness  cri¬ 
terion  was  not  considered  significant  to  this  study.  The 
sampling  plan  described  in  Chapter  III  called  for  selection 
of  alternate  facilities  in  case  permission  could  not  be 
obtained  to  perform  on-site  surveys.  When  AFPRO  General 
Dynamics  refused  access,  the  next  facility  on  the  list  was 
contacted,  and  the  survey  was  conducted  using  a  sample  size 
of  five  facilities.  The  most  significant  limitation  of  this 
study  was  not  the  randomness  feature,  but  rather  the  small 
sample  size.  This  feature,  more  than  the  randomness  crite¬ 
rion,  contributed  to  the  uncontrolled  sampling  error  report¬ 
ed  in  Chapter  III.  The  randomness  criterion  primarily  con¬ 
tributes  to  the  generalizability  of  the  data  obtained  from 
this  study.  Generalizations  possible  from  this  study  are 
discussed  in  the  final  section  of  this  chapter. 

A  second  dimension  of  the  population  addressed  in  the 
survey  design  was  the  requirement  that  the  respondents  were 
administering  the  floor-level  quality  assurance  program  in 
the  plant  representative  office,  and  that  such  respondents 


had  knowledge  of  the  use  and  maintenance  of  contractor 
quality  cost  controls.  This  desire  was  expressed  in  both  an 
introductory  letter  (see  sample  in  appendix  D)  sent  to  each 
plant  representative  office  selected  for  the  study,  and 
during  orientation  meetings  at  the  start  of  each  interview 
session.  All  the  plant  representative  offices  visited  coop¬ 
erated  fully  in  this  regard,  and  a  number  of  different 
personnel  participated  in  the  study  at  each  location.  This 
control  was  used  to  reduce  the  extraneous  variation  typical 
of  ex  post  facto  study  designs. 

A  third  feature  of  the  plan  described  in  Chapter  III 
was  that  interview  questions  would  proceed  from  the  simple 
to  the  complex,  and  from  general  to  specific  items  of  inte¬ 
rest.  This  technique  would  allow  the  interviewer  to  first 
establish  the  respondents'  frames  of  reference,  and  would 
also  limit  the  distortion  effect  of  earlier  questions  on 
later  ones.  Control  was  to  be  accomplished  by  asking  ques¬ 
tions  in  the  order  listed  in  the  interview  instrument  con¬ 
tained  in  appendix  A.  In  the  actual  surveys,  the  only 
deviation  from  this  planned  approach  occurred  when  respon¬ 
dents  indicated  they  were  unfamiliar  with  a  specific  item  of 
interest.  In  such  cases,  these  questions  were  postponed  for 
later  discussion  with  personnel  more  familiar  with  the  item 
of  interest.  This  action  was  in  consonance  with  the  second 
control  feature  described  above,  and  was  therefore  not  con¬ 
sidered  a  detrimental  deviation  from  the  plan.  In  most 
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facilities  visited,  the  postponed  questions  had  to  do  with 
pricing  or  other  contractual  features  generally  beyond  the 
scope  of  floor-level  QA  specialists,  such  as  questions  40, 
41,  47,  48,  52  and  53  (see  appendix  A  for  details). 

A  fourth  feature  of  the  planned  methodology  involved 
techniques  to  obtain  complete  and  accurate  information 
through  respondent  motivation  (13:228).  This  control  was  to 
be  accomplished  by  assuring  respondents  that  their  answers 
would  be  kept  confidential,  and  by  providing  respondents 
with  background  information  about  the  purpose  and  general 
focus  of  the  research  and  the  interviewer's  experience. 
Finally,  motivation  was  to  be  accomplished  by  giving  respon¬ 
dents  the  opportunity  to  compare  their  personal  perspectives 
with  normal  policy.  No  deviation  from  the  planned  motiva¬ 
tion  techniques  occurred  at  any  of  the  facilities  visited. 

A  final  control  feature  of  the  methodology  described  in 
Chapter  III  required  the  interviewer  to  validate  the  data 
obtained  in  the  interview  sessions.  This  control  was  to  be 
accomplished  by  evaluating  supporting  evidence  to  assure  the 
claims  made  by  the  respondents  were  accurate,  and  by  review¬ 
ing  checklists  used  by  government  personnel  to  evaluate  the 
contractors'  process  and  cost  controls.  Of  particular  con¬ 
cern  to  the  researcher  were  claims  regarding  trends  in  the 
area  of  scrap,  rework  and  repair  as  well  as  related  cost 
data.  No  deviations  to  the  plan  occurred,  as  required 
evaluations  were  accomplished  at  each  facility  visited. 


Report  and  Analysis  of  the  Findings 

Bj  summarizing  the  responses  to  the  specific  measure¬ 
ment  questions,  this  section  reports  the  detailed  results  of 
the  interviews  at  the  five  plant  representative  offices.  To 
provide  meaning  to  the  detailed  results,  the  data  from  the 
measurement  questions  were  then  analyzed  in  terms  of  the 
original  investigative  questions.  The  data  were  then  fur¬ 
ther  reduced  to  develop  operational  definitions,  and  the 
relationships  between  process  control,  productivity  and  cost 
were  examined.  This  final  reduction  provided  the  data  to 
resolve  the  original  research  questions,  and  also  provided 
testable  variables  for  follow-on  studies. 

Evaluation  of  Production  and  Manufacturing  Processes. 
As  described  in  Chapter  III,  this  first  group  of  measurement 
questions  was  developed  to  investigate  the  question, 

1.  How  do  DoD  in-plant  personnel  evaluate  contractor 

control  of  production  and  manufacturing  processes? 

This  group  of  measurement  questions  was  subdivided  to  eval¬ 
uate  three  areas  of  process  control:  general  issues,  machin¬ 
ing  and  heat  treating  operations.  The  responses  to  the 
first  eight  "general  issue"  measurement  questions  are  sum¬ 
marized  in  Table  I.  Key  words  from  each  question  are  used 
as  referents  for  the  responses  tabulated.  Where  specific 
follow-on  questions  were  contained  in  the  interview  instru¬ 
ment,  such  as  the  relationship  between  questions  4  and  5, 
the  combined  responses  to  both  questions  are  provided  in  the 


table . 
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TABLE  I 

Summary  of  "General  Issue"  Responses 


Ques.  Referents 


contract  quality 
reqmts 


DoD  surveillance 
manual 


contractor  process 
surveillance 


4  &  5  govt  process 
evaluation 


equip  capability 
and  qualification 

special  operating 
environments 


compliance  with 
work  instructions 


Results 

*  all  5  facilities  had  contracts 
containing  MIL-Q-9858A 

*  3  of  the  5  also  had  contracts 
containing  NHB  5300.4B,  the  NASA 
Quality  Program  specification 

*  2  of  the  5  used  DLAM  8200.1 
exclusively 

*  2  of  the  5  used  AFR  74-1  and 
AFCMDR  178-1  exclusively 

*  1  of  the  5  used  all  three  man¬ 
uals  listed 


*  all  5  facilities  used  in-  n/a 
process  and/or  final  inspection 

*  4  of  the  5  also  used  periodic  n/a 
internal  audits  of  the  processes 

*  all  5  facilities  used  1st  1.0 
piece,  in-process  and  final  in¬ 
spections,  tooling,  further  assy 
checks  (where  applicable),  and 
NDI/NDT 

*  all  5  employed  some  level  of  n/a 
process  audit  to  ensure  compli¬ 
ance  with  required  procedures 

*  3  of  the  5  performed  some  govt  0.6 
evaluation  of  these  elements 

*  only  2  of  the  5  had  reqmts  for  1.0 
environmental  controls  —  each 

of  these  used  govt  audit  to  ver¬ 
ify  compliance 

*  2  of  the  5  used  a  combination  1.0 
of  product  inspection  and  audit 

to  verify  this  reqmt,  and  the 
remaining  3  facilities  relied 
solely  on  audits  to  verify  this 
reqmt 


Rating  * 


*  Ratings  represent  the  percentage  of  positive  responses. 
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The  last  two  "general  issue"  questions,  numbered  9  and 
10  on  the  interview  instrument,  obtained  the  respondents' 
definitions  of  process  control  and  process  yield.  A  compo¬ 
site  definition,  based  on  repetitive  elements  in  each  re¬ 
spondent's  answer  is  reported  below: 

Process  Control  is  an  assessment  of  the  adequacy  of  and 
compliance  with  procedures  and  work  instructions  that 
control  the  elements  of  the  process  that  affect  product 
quality . 

Process  Yield  is  a  measure  of  the  efficiency  of  the 
process  in  terms  of  output  of  product  that  meets  speci¬ 
fication  requirements. 

Later  in  this  section,  these  definitions  and  those 
provided  in  Chapter  I  will  provide  a  frame  of  reference  for 
the  development  of  the  operational  definitions  proposed  for 
use  in  follow-on  studies.  In  addition,  the  "rating"  column 
in  this  and  other  tables  will  be  used  in  the  operational 
definitions  to  evaluate  the  original  research  questions. 

The  second  group  of  measurement  questions  associated 
with  the  first  investigative  question  is  summarized  in  Table 
II.  This  table  covers  questions  11  through  23  on  the  inter¬ 
view  checklist,  which  were  referred  to  as  the  "machining 
operation"  questions  in  the  preceding  chapter.  The  final 
series  of  measurement  questions  associated  with  this  first 
investigative  question  were  identified  as  "heat  treating 
operation"  questions.  This  element  of  process  control  was 
addressed  by  questions  24  through  31  on  the  interview  check¬ 
list.  The  responses  to  this  final  subgroup  of  questions  are 


summarized  in  Table  III. 


TABLE  II 


Summary  of  "Machining  Operation"  Responses 


Ques 

i. 

Referents 

Results 

Rating 

11 

basic  machining 

*  all  5  facilities  had  some  form 

n/a 

operations 

of  metal  removing  operation 
♦  4  of  the  5  had  some  type  of 
shaping  operation 

n/a 

12  & 

13 

equip  maint  and 

*  4  of  the  5  had  some  govt  eval¬ 

0.8 

calibration 

uation  of  maint  records 

*  4  of  the  5  had  govt  evaluation 
of  calibration  records,  with  1 

0.8 

of  the  5  witnessing  the  calib. 

14  & 

15 

tooling/ fixture 

*4  of  the  5  contractors  were 

n/a 

qualification. 

required  to  check  for  tool  wear 

calib.  and  wear 

and  calib  based  on  usage  or  time 
*  4  of  the  5  had  periodic  govt 
evaluation  of  tooling  qualifica¬ 
tion  and  calibration 

1.0 

16 

lubricants  and 

*  2  of  the  5  had  some  govt  eval¬ 

0.4 

cooling  oils 

uation  of  lubricants  as  part  of 
equip  maint 

17 

set-up,  feeds  and 

♦  2  of  the  5  had  some  govt  eval¬ 

0.4 

speeds 

uation  of  these  elements 

18 

inserts  &  offsets 

*  none  of  the  facilities  had  any 
govt  evaluation  of  this  element 

0.0 

19 

depth  of  cut 

*  1  of  the  5  had  some  govt  veri¬ 
fication  of  this  element 

0.2 

20  & 

21 

statistical  ana¬ 

*  4  of  the  5  contractors  were 

0.5 

lysis 

required  to  use  statistical  con¬ 
trol  and  2  of  these  4  had  some 

govt  evaluation  of  the  contrac¬ 
tor  data 

22  & 

23 

advanced  mach 

*  all  5  used  N/C  equip 

n/a 

techniques 

*  2  of  the  5  used  CAD/CAM 

n/a 

*  2  of  the  5  used  robotics  and 

n/a 

integrated  gaging 
*  1  of  the  5  had  govt  evaluation 
of  N/C  tapes,  and  1  of  the  5  had 

0.4 

inspection  of  hardware  made  on 
N/C  equip 
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TABLE  III 


Summary  of  "Heat  Treating  Operation"  Responses 


Ques.  Referents 
24  type  of  heat  treat 


25  preparations  for 
heat  treating 

26  rate  of  heating 
and  soak  time 


27  quenching 


28  annealing 


29  contractor  test¬ 
ing 


30  &  31  govt  verification 
of  contractor 
testing 


Results 

*  3  of  the  5  treated  both  steel 
and  aluminum 

*  1  of  the  5  treated  only  steel 

*  1  of  the  5  treated  only  alum¬ 
inum 

*  only  1  facility  had  any  govt 
verification  of  the  preparations 

*  4  of  the  5  had  govt  verifica¬ 
tion  of  the  contractor  records 
of  time,  temp  and  pressure 

*  2  of  the  5  had  govt  evaluation 
of  this  element 

*  only  2  of  the  5  contractors 
were  annealing  parts,  and  only  1 
govt  activity  was  verifying  the 
contractor  controls 

*  4  of  the  5  contractors  used 
lab  testing,  with  the  5th  rely¬ 
ing  solely  on  hardness  tests 

*  all  5  facilities  had  some  form 
of  govt  evaluation  of  contractor 
test  records,  and  2  of  the  5  ac¬ 
tually  witnessed  some  of  the 
tests 


In  general,  the  answer  to  the  £irst  investigative  ques¬ 
tion  is  a  composite  of  the  responses  summarized  in  the  first 
three  tables  of  this  section.  Table  I  addressed  ''general 
issues"  associated  with  process  control.  Since  all  the 
facilities  visited  contained  requirements  for  a  MIL-Q-9858A 
Quality  Program  or  its  NASA  NHB  5300. 4B  equivalent  (see 
question  1),  the  contractors'  process  controls  should  encom¬ 
pass  all  the  elements  listed  in  Chapter  II  of  this  report, 
as  examined  by  the  individual  measurement  questions.  This 
was  generally  true,  with  the  exception  of  the  one  facility 
that  did  not  perform  internal  audits  (see  question  3).  The 
government  surveillance  of  such  programs  should  also  be 
similar,  except  as  permitted  by  peculiar  DoD  manuals  (see 
question  2).  However,  the  responses  to  questions  4  through 
8  indicated  this  was  not  the  case.  The  difference  in  Gov¬ 
ernment  surveillance  levels  becomes  even  more  obvious  as  the 
results  of  Tables  II  and  III  are  evaluated. 

Table  II  addressed  specific  process  controls  normally 
associated  with  "machining  operations."  The  individual 
measurement  questions  were  based  on  process  controls  identi¬ 
fied  in  Chapters  II  and  III  of  this  report.  Since  most  of 
the  contractors  used  similar  machining  operations  (see  ques¬ 
tions  11  and  22),  and  had  similar  control  for  tooling  and 
fixture  wear  (see  question  14)  and  statistical  analysis  (see 
question  20),  the  level  of  government  surveillance  should  be 


similar.  However,  the  responses  to  questions  12,  13,  15-19, 
21  and  23  indicate  a  wide  range  in  the  level  of  government 
surveillance  of  the  cited  process  controls. 


Table  III  addressed  specific  process  controls  normally 
associated  with  "heat  treating  operations."  These  indivi¬ 
dual  measurement  questions  were  also  based  on  typical  pro¬ 
cess  controls  identified  in  Chanters  II  and  III.  Again,  the 
contractor  controls  were  generally  consistent  (see  questions 
24  and  29);  but,  as  demonstrated  in  the  responses  to  ques¬ 
tions  25-28,  30  and  31,  the  level  of  government  surveillance 
of  these  control  elements  again  differed. 

Because  of  the  varying  levels  of  government  surveil- 

t 

lance,  the  overall  response  to  this  first  investigative 
question  must  be  qualified.  While  the  major  aerospace  fa¬ 
cilities  visited  manufactured  quite  different  products,  the 
contractors  generally  employ  similar  process  controls  to 
meet  the  requirements  of  MIL-Q-9858A  and/or  NHB  5300. 4B. 
The  government  personnel  responsible  for  evaluating  these 
process  controls  generally  perform  some  surveillance,  but 
the  individual  process  control  elements  evaluated  appear  to 
be  at  the  discretion  of  the  government  personnel  assigned  to 
each  facility.  This  difference  will  be  discussed  in  more 
detail  in  Chapter  V. 


fi? 


Evaluation  of  the  Contractor  Cost  Controls.  This  group 
of  measurement  questions  vas  designed  to  investigate  the 
following  question, 

2.  How  do  DoD  in-plant  personnel  evaluate  the  contrac¬ 
tor's  cost  controls? 

The  responses  to  questions  33  through  39  provide  the  data  to 
resolve  this  question.  These  responses  are  summarized  in 
Table  IV.  In  addition,  a  composite  of  the  respondents' 
definitions  of  nonconforming  material,  provided  in  response 
to  question  32  on  the  interview  instrument,  is  reported 
below: 

Nonconforming  Material  is  any  product  that  does  not 
conform  to  specification,  drawing  or  contract  require¬ 
ments. 

As  noted  in  Chapter  III,  nonconforming  material  is 
correlated  with  cost  control  in  the  government  specifica¬ 
tions  cited  in  this  report  (9;  35).  When  compared  with  the 
other  data  reported  in  this  section,  this  composite  defini¬ 
tion  of  nonconforming  material  will  contribute  to  the  devel¬ 
opment  of  the  second  operational  definition  proposed  for  use 
in  follow-on  studies.  The  actual  operational  definitions 
are  contained  in  a  later  subsection  of  this  chapter. 

As  above,  the  answer  to  this  question  is  a  composite  of 
the  responses  summarized  in  Table  IV.  In  general,  the 
contractors  had  similar  requirements  for  the  control  of 
nonconforming  material  and  the  cost  of  quality  (see  question 
33).  Most  of  the  contractors  also  performed  some  trend 
analysis  of  their  nonconforming  material  and  cost  of  quality 
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TABLE  IV 


Ques. 

33 

34 

35  &  36 

37 

38 

39 


Summary  of  Cost  Control  Responses 


Referents 


Results 


contract  provisions  *  all  5  facilities  had  reqmts 

for  maintenance  of  cost  of  qual¬ 
ity  data  under  MIL-Q-9858A 

*  2  of  the  5  had  nonconforming 
material  per  MIL-STD-1520A 

*  1  of  the  5  had  nonconforming 
material  per  DLAM  8200.1 

*  2  of  the  5  used  MIL-STB-1520A 
or  DLAM  8200.1  depending  on  the 
program 

govt  role  on  re-  *  when  MIL-STD-1520A  applied, 
work,  repair  and  the  contractor  dispositioned  re¬ 
scrap  work,  scrap  and  standard  repairs 

w/o  prior  govt  approval 

*  when  DLAM  8200.1  applied,  the 
govt  provided  prior  approval  on 
all  MRB  actions 


trend  analysis  ♦  4  of  the  5  had  contractor  data 

available,  and  the  govt  perform¬ 
ed  some  review  of  the  data 


product  quality 
and  sampling 


quality  cost 
analysis 


overall  discrep¬ 
ancy  costs 


*  only  1  contractor  regularly 
used  samples  based  on  prior  his¬ 
tory;  the  others  performed  pri¬ 
marily  100%  inspection,  due  to 
contract  reqmts  or  small  lots 

*  4  of  the  5  facilities  had  some 
govt  analysis  of  contractor  cost 
data,  and  the  5th  facility  had 
the  MIL-Q-9858A  reqmts  waived  by 
the  PC0 

*  2  of  the  5  had  govt  evaluation 
of  replacement  costs 

*  3  of  the  5  had  govt  evaluation 
of  reinspection  costs 

*  1  of  the  5  had  govt  evaluation 
of  repair/overhaul  data  only 

*  1  of  the  5  had  the  MIL-Q-9858A 
reqmts  waived  by  the  PC0 


data  (see  question  35).  Only  in  the  areas  of  reduced  in¬ 
spection  (see  question  37)  did  the  contractor  approaches 
truly  differ,  but  this  difference  was  generally  due  to 
contract  requirements  and/or  small  lot  sizes  of  production 
hardware. 

The  level  of  government  surveillance  of  contractor  cost 
controls,  as  demonstrated  by  the  responses  to  questions  34, 
36,  38  and  39,  was  much  more  consistent  than  that  reported 
for  process  controls.  The  only  real  variance  occurred  in 
the  area  of  evaluating  overall  discrepancy  costs  (see  ques¬ 
tion  39),  but  the  majority  still  performed  some  surveillance 
of  this  control  element.  The  response  to  this  second  inves¬ 
tigative  question  is:  government  QA  personnel  generally 
assess  all  the  elements  of  cost  control  identified  in  chap¬ 
ters  II  and  III. 

Relationships  Among  Process  Controls.  Cost  and  Produc¬ 
tivity.  The  third  group  of  measurement  questions  developed 
in  Chapter  III  was  designed  to  investigate  the  question, 

3.  If  the  DoD  in-plant  personnel  evaluate  the  process 
controls,  how  do  they  define  a  relationship  between 
such  controls  and  productivity  or  cost? 

The  responses  to  the  measurement  questions  numbered  40 
through  47  provide  the  data  to  resolve  the  third  investiga¬ 
tive  question.  The  results  are  summarized  in  Table  V.  This 
data  will  be  combined  with  that  developed  in  response  to  the 
fourth  investigative  question  (reported  in  the  next  subsec¬ 
tion)  to  develop  the  last  two  operational  definitions. 


TABLE  V 


Summary  of  Relationship  Data  Responses 


Ques. 

Referents 

Results 

Rating 

40 

ManTech  and  TechMod 

*  4  of  the  5  contractors  were 

n/a 

funding 

receiving  funds  for  one  or  both 
of  these  programs,  which  result¬ 

ed  in  improved  capital  equip 
*  the  5th  facility  received  no 

n/a 

special  funding,  but  did  receive 
new  capital  equip  as  GFP 

41 

use  of  capital 

*  only  2  of  the  5  have  document¬ 

0.4 

equip 

ed  productivity  improvements  re¬ 
lated  to  new  capital  equip 

42,  43 

corrective  action 

*  in  2  of  the  5,  the  govt  re¬ 

0.8 

&  44 

and  productivity 

ported  effective  C/A  reduced  the 
levels  of  nonconforming  material 
*  2  of  the  5  reported  short-term 

n/a 

improvements,  but  no  long-term 
reductions 

*  the  5th  facility  reported  very 
little  C/A  due  to  low  rates  of 
new  production  hardware 

n/a 

45 

rework  and  repair 

*  3  of  the  5  reported  reductions 

0.6 

related  to  produc¬ 
tivity 

in  rework  and  repair 
*  1  of  the  5  reported  little  re¬ 

n/a 

work  and  repair  due  to  low  rates 
of  new  production 
*  the  5th  facility  had  no  govt 
review  of  rework  and  repair  data 

n/a 

46 

productivity 

*  4  of  the  5  reported  overall 

0.8 

trends 

productivity  improvements 
*  the  5th  facility  had  no  govt 
data  on  which  to  evaluate  trends 

n/a 

47 

effect  of  process 

*  in  4  of  the  5,  the  govt  noted 

0.8 

control  and  produc¬ 

reduced  costs  due  to  productiv¬ 

tivity  on  costs 

ity  improvements  in  mach  and  H/T 

areas 

*  in  the  5th  facility,  the  govt 
was  unaware  of  any  cost  data  to 
support  or  refute  improvement  in 
the  mach  and  H/T  areas 

n/a 
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As  was  done  with  the  preceding  investigative  questions, 
this  question  will  be  resolved  by  a  composite  response  based 
on  data  contained  in  Table  V.  While  all  the  contractors 
were  receiving  funds  to  procure  new  capital  equipment  (see 
question  40),  the  government  was  only  aware  of  documented 
productivity  improvements  associated  with  capital  equipment 
in  two  of  the  contractor  facilities  (see  question  41). 

Using  the  definitions  of  yield  and  nonconforming  mater¬ 
ial  developed  above,  the  effect  of  corrective  action  on  the 
causes  and  the  levels  of  nonconforming  material  (see  ques¬ 
tions  42-44)  was  not  considered  a  long  term  contributor  to 

increased  productivity  or  yield  in  most  of  the  facilities. 

* 

The  effect  of  the  contractors*  efforts  to  control  rework  and 
repair  levels  (see  question  45)  was  considered  a  contributor 
to  increased  productivity  or  yield  at  most  of  the  facili¬ 
ties.  Government  personnel  generally  found  the  overall 
productivity  trends  and  cost  data  for  the  machining  and  heat 
treating  operations  (see  questions  46  and  47)  to  be  improv¬ 
ing.  As  a  result,  the  response  to  the  third  investigative 
question  indicates  a  positive  relationship  among  process 
controls,  productivity  and  cost  are  probable.  This  finding 
will  be  further  evaluated  in  a  later  subsection,  using  the 
data  from  the  fourth  investigative  question  and  a  cross¬ 
classification  table. 


measurement  questions  was  designed  to  investigate  the  ques¬ 
tion, 


4.  What  data  is  available  to  demonstrate  relationships 
among  process  controls,  productivity  improvement  and 
reduced  costs? 

The  responses  to  this  group  of  measurement  questions,  num¬ 
bered  48  through  53,  are  summarized  in  Table  VI.  As  noted 
in  the  preceding  subsection,  the  data  from  these  questions 
will  be  combined  with  that  data  summarized  in  Table  V  to 
develop  the  last  two  operational  definitions.  The  data  in 
these  last  two  subsections  will  ultimately  provide  the  re¬ 
sults  necessary  to  address  the  shortcomings  of  the  present 
literature  so  often  cited  in  this  report.  This  point  will 
be  discussed  at  length  in  Chapter  V. 

Like  the  others,  the  answer  to  this  investigative  ques¬ 
tion  is  a  composite  of  the  responses  summarized  in  Table  VI. 
As  before,  similar  circumstances  existed  for  each  of  the 
contractors  (see  questions  48-50).  All  five  contractors 
maintain  detailed  cost  and  trend  data,  and  such  data  is  made 
available  for  government  review.  However,  the  contractors 
are  apparently  not  provided  contractual  incentives  to  in¬ 
crease  productivity,  even  though  the  literature  reviewed  in 
Chapter  II  indicated  such  provisions  should  be  considered 


when  developing  Tech  Mod  programs  (10:121). 


TABLE  VI 


Summary  of  Supporting  Bata  Responses 


Ques.  Referents  Results 

48  contract  incentives  *  none  reported  by  govt 

to  improve  produc¬ 
tivity 

49  &  50  cost  and  trend  *  all  5  contractors  collected 

data  detailed  cost  and  trend  data 

*  only  2  of  the  5  were  contract¬ 
ually  required  to  deliver  copies 
of  the  data  to  the  govt,  but  all 
five  made  the  data  available  for 
govt  review 

*  4  of  the  5  govt  offices  verify 
the  contractor  maintains  re¬ 
quired  data,  but  only  1  office 
considered  their  review  a  vali¬ 
dation  of  that  data 

52  &  53  productivity  *  1  of  the  5  govt  activities  re¬ 

improvements  and  ported  a  price  reduction  due  to 
reduced  costs  data  on  productivity  improvement 

*  3  of  the  5  govt  activities  had 
data  on  the  mach  and  H/T  areas, 
but  such  data  was  not  evaluated 
unless  threshholds  were  exceeded 

*  all  5  govt  activities  evaluate 
direct  cost  and  overhead;  in  all 
but  one  facility,  reductions 
were  based  on  historical  con¬ 
tractor  data,  and  did  not  in¬ 
clude  an  analysis  of  productiv¬ 
ity  data 


51  validity  and 

accuracy  of  data 


The  government  surveillance  levels  and  use  of  available 


data,  as  demonstrated  in  the  responses  to  questions  51-53, 
varied  considerably.  While  most  of  the  government  QA  per¬ 
sonnel  did  verify  the  contractor  maintained  the  required 
data,  only  one  of  the  plant  representative  offices  attempted 
to  validate  the  accuracy  of  the  data.  Most  of  the  Govern¬ 
ment  offices  had  data  available  on  costs  in  the  machining 
and  heat  treating  areas,'  but  only  one  of  the  offices  report¬ 
ed  a  price  reduction  directly  associated  with  productivity 
improvements.  And  finally,  while  all  the  government  activi¬ 
ties  evaluated  related  costs  such  as  direct  labor  and  over¬ 
head,  only  one  Included  an  analysis  of  productivity  to 
determine  whether  cost  savings  were  appropriate. 

As  a  result  of  the  foregoing  considerations,  the  answer 
to  this  final  investigative  question  must  also  be  qualified. 
While  the  data  supports  a  relationship  between  process  con¬ 
trols  and  productivity  improvement,  the  responses  to  this 
group  of  questions  do  not  support  a  relationship  between 
these  factors  and  reduced  weapon  system  cost.  Further  ana¬ 
lysis  of  this  issue  occurs  below,  when  the  relationships  are 
examined  in  terms  of  the  original  research  questions. 

A  final  group  of  questions  in  the  survey  instrument 
sought  to  determine  personal  data  about  the  respondents. 
However,  as  noted  in  Chapter  III,  this  data  was  only  intend¬ 
ed  for  follow-up  by  the  researcher.  As  such,  no  summaries 
of  the  personal  data  will  be  provided  in  this  report. 
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Development  of  Operational  Definitions.  As  noted  in 
Chapter  III,  operational  definitions  "should  specify  charac¬ 
teristics  to  be  observed  and  how  they  are  to  be  observed 
[11:28]."  The  four  operational  definitions,  or  variables, 
called  for  in  Chapter  III  were:  (1)  adequate  process  con¬ 
trols;  (2)  adequate  cost  controls;  (3)  improved  productiv¬ 
ity;  and  (4)  associated  cost  reductions.  Based  on  the  data 
contained  in  Tables  I  through  VI,  and  the  analyses  provided 
in  the  preceding  subsections,  the  following  operational 
definitions  will  be  used  to  determine  whether  relationships 
exist  between  process  controls,  productivity  and  reduced 
weapon  system  costs.  As  noted  in  Chapter  III,  some  informa¬ 
tion  is  lost  when  data  are  reduced  and  manipulated,  especi¬ 
ally  when  using  small  sample  sizes.  However,  the  only  way 
to  add  meaning  to  the  large  amount  of  data  contained  in 
Tables  I  through  VI  is  to  reduce  it  to  categories  that 
facilitate  analysis. 

Adequate  Process  Controls.  This  characteristic  is 
based  on  the  composite  response  to  the  first  investigative 
question.  The  characteristic  should  be  observed  in  the 
fashion  described  in  the  preceding  analysis;  that  is,  by 
comparing  the  process  control  requirements  levied  on  the 
contractors  with  the  level  of  surveillance  performed  by  the 
cognizant  government  plant  representative  offices.  This 
approach  is  consistent  with  the  composite  definition  of 
"process  control"  provided  earlier  in  this  section. 
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Where  similar  requirements  are  levied  on  the  majority 
of  the  contractors,  the  adequacy  of  the  process  controls  is 
determined  by  subjective  and  widely  varying  evaluations  by 
the  government  personnel,  but  at  least  some  of  the  listed 
control  elements  are  evaluated  at  all  of  the  plant  represen¬ 
tative  offices.  The  value  assigned  this  characteristic  is 
calculated  by  averaging  the  percentage  of  plant  representa¬ 
tive  offices  that  evaluated  the  elements  indicated  by  ques¬ 
tions  4-8,  12,  13,  15-19,  21,  23,  25-28,  30  and  31.  This 
percentage  is  indicated  by  the  "rating"  assigned  each  ques¬ 
tion  in  the  various  tables.  Equation  (2)  will  simplify  this 
calculation: 

APC  -  (£x)/n  (2) 

where:  APC  ■  adequate  process  controls 

x  ■  ratings  assigned  Individual  questions 
n  «  number  of  ratings  given 

Using  the  data  from  Tables  I  through  III  in  equation 
(2),  the  value  for  this  characteristic  is: 

APC  -  ll.U/18  =  0.61 

Since  questions  5  and  31  follow-up  questions  4  and  30  re¬ 
spectively,  only  18  elements  are  represented  in  this  calcu¬ 
lation.  In  addition,  where  less  than  five  contractors  per¬ 


formed  a  specific  function  (as  indicated  by  the  responses  to 
questions  7,  14/15,  20/21,  and  28),  the  rating  assigned  that 
element  is  based  on  the  percentage  of  contractors  performing 


the  function 


The  APC  value  calculated  above  will  be  used 


again  in  the  cross-classification  tables  developed  in  the 
next  subsection. 

Adequate  Cost  Controls.  This  characteristic  is 
based  upon  the  composite  response  to  the  second  investiga¬ 
tive  question.  It  too  should  be  observed  in  the  fashion 
described  in  the  preceding  analysis;  that  is,  by  comparing 
the  cost  control  requirements  levied  on  the  contractors  with 
the  level  of  surveillance  performed  by  the  government  plant 
representative  offices.  Where  similar  requirements  and 
levels  of  data  analysis  apply  to  the  majority  of  the  con¬ 
tractors,  the  adequacy  of  the  cost  controls  is  determined  by 
government  surveillance  of  all  of  the  control  elements  list¬ 
ed  at  the  majority  of  the  facilities  visited.  This  approach 
is  consistent  with  the  composite  definition  of  "nonconform¬ 
ing  material"  provided  earlier  in  this  section,  in  that  the 
operational  definition  stresses  those  activities  necessary 
to  verify  contractors'  efforts  to  control  nonconforming 
material . 

The  value  assigned  this  characteristic  is  based  on  the 
average  number  of  plant  representative  offices  that  evalu¬ 
ated  the  cost  control  elements  listed  in  questions  34,  36, 
38  and  39.  This  percentage  is  indicated  by  the  "rating" 
assigned  each  question  in  Table  IV.  Equation  (3)  will 
simplify  this  calculation: 


ACC  -  (£y)/n 


(3) 


where:  ACC  -  adequate  cost  controls 

y  »  rating  assigned  questions  34,  36,  38  and  39 
n  m  number  of  ratings  given 

Using  the  data  from  Table  IV  in  equation  (3),  the  value 
for  this  characteristic  is: 

ACC  -  4.05/5  -  0.81 

For  question  34,  positive  action  was  assured  because  all 
five  facilities  had  acceptable  government  evaluation  regard¬ 
less  of  which  specification  applied.  The  ratings  assigned 
questions  38  and  39  are  based  on  the  number  of  contractors 
required  to  maintain  such  data.  The  ACC  value  calculated 
above  will  also  be  used  in  the  cross-classification  tables 
developed  in  the  next  subsection. 

Improved  Productivity.  This  characteristic  is 
based  upon  the  composite  response  to  the  third  and  fourth 
investigative  questions.  It  is  observed  by  comparing  the 
factors  of  productivity  improvement  with  the  supporting  data 
analysis.  Improved  productivity  will  be  deemed  to  have 
occurred  if  the  claimed  trends  in  the  productivity  factors 
evaluated  are  improving  based  on  actual  government  valida¬ 
tion  of  the  data.  The  value  assigned  this  characteristic  is 
calculated  by  averaging  the  number  of  facilities  that 
claimed  improvements  for  the  factors  identified  in  questions 
41-46,  then  weighting  this  percentage  by  the  percentage  of 

74 


the  positive  responses  to  question  51,  where  a  positive 
response  means  that  validation  has  occurred.  Equation  (4) 
will  simplify  this  calculation: 

IP  -  a[(£z)/n]  (4) 

where:  IP  »  improved  productivity 

z  •  ratings  assigned  to  questions  41-46 

a  -  rating  assigned  to  question  51 

n  ■  number  of  improvement  factor  ratings  given 

Using  the  data  from  Tables  V  and  VI  in  equation  (4), 
the  value  for  this  characteristic  is: 

IP  -  (0.2)(2.6/4)  -  (0.2)(0.65)  -  0.13 

The  IP  value  calculated  above  is  sensitive  to  both  the  level 
of  claims  made  regarding  improved  productivity  and  the  a- 
mount  of  government  validation  of  such  claims.  Since  ques¬ 
tions  42,  43  and  44  were  all  interrelated,  only  four  ele¬ 
ments  were  considered  in  developing  the  number  of  improve¬ 
ment  factors.  As  will  be  done  with  the  values  from  the 
preceding  operational  definitions,  the  IP  value  calculated 
above  will  be  used  in  the  cross-classification  tables  in  the 
next  subsection. 

Associated  Cost  Reductions.  This  characteristic 
is  also  based  on  the  composite  responses  to  the  third  and 
fourth  investigative  questions.  It  is  observed  by  comparing 
claimed  cost  reductions  with  the  supporting  data  analysis. 
Associated  cost  reductions  will  be  deemed  to  have  occurred 
if  the  claimed  cost  savings  are  shown  to  have  resulted  in 
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reduced  costs  to  the  government.  The  claimed  cost  savings 
were  based  on  data  provided  to  the  government  by  the  con¬ 
tractor  machining  and  heat  treating  departments.  This  data 
must  have  been  subjected  to  validation  by  on-site  government 
personnel,  and  shown  to  have  resulted  in  reduced  costs  as 
verified  by  this  researcher.  The  value  assigned  this  char¬ 
acteristic  is  calculated  by  weighting  the  percentage  "rat¬ 
ing"  of  facilities  responding  positively  to  question  47  with 
the  average  of  positive  responses  to  questions  51,  52  and 
53.  Equation  (5)  will  simplify  this  calculation: 

ACR  -  b[(£c)/n]  (5) 

where:  ACR  »  associated  cost  reductions 

b  ■  rating  assigned  to  question  47 
c  ■  ratings  assigned  to  questions  51,  52  and  53 
n  »  number  of  supporting  data  ratings  given 

Using  the  data  from  Tables  V  and  VI  in  equation  (5), 
the  value  for  this  characteristic  is: 

ACR  -  0.8(0. 6/3)  -  0.8(0. 2)  -  0.16 

Like  the  preceding  IP  calculation,  the  ACR  value  calculated 
above  is  sensitive  to  the  level  of  claimed  cost  savings  and 
the  amount  of  government  validation  of  such  claims.  In  this 
instance,  three  ratings  were  considered  to  develop  the  sup¬ 
port  data  average.  The  first  two  categories  of  responses  to 
questions  52  and  53  actually  addressed  the  same  issue,  and 
were  thus  considered  as  one  combined  response.  The  respon¬ 
ses  to  question  51  and  the  third  category  under  qestions 


52/53  made  up  the  remainder  of  the  supporting  data  average. 
Like  the  other  three  operational  definition  values  calcu¬ 
lated  in  this  subsection,  the  ACR  value  calculated  above 
will  be  used  in  the  cross-classification  tables  developed  in 
the  next  subsection. 

The  respondents  were  asked  to  consider  their  defini¬ 
tions  of  "yield”  and  "nonconforming  material"  (provided 
above)  when  answering  the  questions  involved  in  developing 
the  last  two  operational  definitions.  This  approach  assured 
these  operational  definitions  were  consistent  with  the  defi¬ 
nitions  of  "productivity"  and  "yield"  provided  in  Chapter  I 
of  this  report.  In  light  of  that  consistency,  the  next 
subsection  will  examine  the  relationships  described  by  these 
operational  definitions  in  terms  of  the  original  research 
questions. 

Relationships  Examined  in  Terms  of  the  Original  Re¬ 
search  Questions.  As  noted  at  the  beginning  of  this  sec¬ 
tion,  the  categories  developed  in  analyzing  the  four  inves¬ 
tigative  questions  above  could  be  used  to  test  whether 
relationships  exist  between  process  controls,  productivity 
improvements  and  reduced  weapon  system  costs.  This  testing 
will  be  accomplished  by  a  final  data  reduction,  in  which  the 
preceding  operational  definitions  are  cross  classified  as 
shown  in  Tables  VII  and  VIII. 


TABLE  VII 


Relationship  Between  Process  Controls  and 
Productivity  Improvement 


Process  Control  Productivity  Improvement 
Value  Value 

APC  0.61 
ACC  0.81 

IP  —  0.13 

Table  VII  compares  the  values  derived  from  the  preced¬ 
ing  operational  definitions,  to  determine  whether  a  rela¬ 
tionship  exists  between  process  controls  (including  cost 
controls,  as  correlated  in  Chapter  II)  and  productivity 
improvements.  This  tabular  comparison  is  intended  to  clari¬ 
fy  such  relationships,  and  to  resolve  the  first  research 
question , 

1.  Are  DoD  in-plant  personnel  aware  of  a  relationship 
between  quality  assurance  process  controls  and  produc¬ 
tivity? 

While  the  analysis  in  the  preceding  subsections  might 
lead  to  the  belief  that  DoD  in-plant  personnel  are  doing  a 
good  job  evaluating  process  and  cost  controls,  with  a  resul¬ 
tant  improvement  in  productivity.  Table  VII  shows  a  poten¬ 
tial  weakness  in  the  current  DoD  approach  to  contractor 
surveillance.  The  adequacy  of  process  control  (APC)  and 
adequacy  of  cost  control  (ACC)  values  indicate  the  majority 
of  the  plant  representative  offices  are  assuring  contractors 
control  these  elements.  In  addition,  the  DoD  in-plant  per¬ 


sonnel  indicated  there  was  an  a  productivity  increase  in  the 


plants  which  had  adequate  process  and  cost  controls.  How¬ 
ever,  when  the  increase  in  productivity  is  weighted  by  the 
amount  of  DoD  validation  of  contractor  data,  as  indicated  by 
the  improved  productivity  (IP)  value,  it  might  be  concluded 
that  claimed  productivity  increases  are  tentative  at  best. 
Table  VII  indicates  that  less  than  one  of  every  five  (0.13 
to  be  exact)  facilities  would  apparently  be  able  to  demon¬ 
strate  a  relationship  between  adequate  process  (and  cost) 
controls  and  improved  productivity. 

TABLE  VIII 

Relationship  Between  Process  Controls  and 
Reduced  Weapon  System  Costs 

Process  Control  Reduced  Cost 
Value  Value 

APC  0.61 

ACC  0.81 

ACR  —  0.16 

Table  VIII  also  compares  the  values  derived  from  the 
operational  definitions  to  determine  whether  a  relationship 
exists  between  the  process  controls  and  reduced  weapon  sys¬ 
tem  costs.  This  tabular  comparison  is  made  to  clarify  such 
relationships,  and  to  resolve  the  second  research  question, 

2.  If  such  a  relationship  exists,  are  associated  cost 
savings  passed  on  to  DoD  in  future  weapon  system  pro¬ 
curements? 


The  analysis  of  Table  VII  might  lead  to  the  conclusion 
that  further  analysis  of  the  second  research  question  is 
impractical.  However,  the  standard  error  associated  with 
the  preceding  data  (to  be  reviewed  in  the  next  section) 
indicates  this  research  question  should  also  be  evaluated. 
Like  the  preceding  analysis  of  the  relationships  between 
process  controls  and  productivity  improvement,  the  APC  and 
ACC  operational  definition  values  used  in  Table  VIII  indi¬ 
cate  the  contractors  are  controlling  the  processes  and  their 
associated  costs.  In  addition,  the  DoD  in-plant  personnel 
indicated  cost  savings  were  occurring  in  the  heat  treating 
and  machining  areas.  But,  the  associated  cost  reduction 
(ACR)  value  is  weighted  by  th»  amount  of  DoD  validation  of 
the  contractor  data  and  proof  of  reduced  costs.  This  analy¬ 
sis  indicates  there  is  little  likelihood  (i.e.,  in  less  than 
one  of  every  five  facilities,  or  0.16)  of  cost  savings 
resulting  from  adequate  process  controls  being  passed  on  to 
DoD  in  future  weapon  system  procurements. 

The  preceding  analysis  takes  a  rather  narrow  view  of 
the  relationships  that  might  occur  among  process  controls, 
productivity  and  weapon  system  costs.  But,  the  purpose  of 
an  empirical  study  is  to  collect  verifiable  data,  and  the 
methodology  employed  in  the  instant  study  assures  this  pur¬ 
pose  was  met.  Chapter  V  will  identify  some  of  the  methods 
DoD  in-plant  personnel  can  use  to  overcome  the  weaknesses  in 
their  present  surveillance  methods. 


Statistical  Tests 


One  of  the  assumptions  identified  in  Chapter  III  was 
that  the  sampling  error  associated  with  the  small  sample 
size  used  for  this  study  could  not  be  controlled.  As  noted 
in  Chapter  III,  this  uncontrolled  situation  would  exist  for 
a  variety  of  sample  intervals  and  confidence  levels.  Equa¬ 
tion  (1),  the  formula  used  to  determine  the  standard  error, 
is  restated  below: 

std  error  -  [ pq/(n-l ) ] l/2[ (N-n)/(N-l) ] */2 

where:  p  »  proportion  of  population  having  an  attribute 

q  »  proportion  of  population  not  having  an  attribute 
N  »  size  of  population 
n  a  size  of  sample 

Using  this  study's  population  size  of  43,  a  sample  size 
of  five,  and  the  value  associated  with  the  level  of  govern¬ 
ment  validation  of  contractor  data  (question  51),  the  stan¬ 
dard  error  associated  with  one  of  the  "rating"  proportions 
used  in  this  study  is  calculated  as  follows: 

std  error  -  { [ (0. 2) (0 . 8 ) ] /( 5-1 ) } 1/2 [ (43-5) /( 43-1 ) ] 1/2 

-  (0. 16)(0.951) 

-  +/-  0.152 

The  same  type  of  calculation  can  be  made  using  the 
rating  proportion  associated  with  any  other  question  used  in 
this  survey.  For  example,  the  standard  error  associated 
with  the  rating  (0.6)  assigned  to  question  45  would  be 
calculated  as  follows: 


81 


std  error  -  { [ (0.6) (0. 4) ]/(5-l ) } l/2[ (43-5)/(43-l ) ] 1/2 

-  (0.24)(0.951) 

-  +/-  0.228 


The  standard  errors  calculated  above  are  just  two  of  32 
such  calculations  that  would  be  required  to  assess  the 
accuracy  of  the  operational  definition  values  developed  in 


this  chapter.  The  standard  errors  associated  with  each  of 
the  four  rating  proportions  used  to  calculate  the  ACR  opera¬ 
tional  definition  would  result  in  a  variety  of  possible 
values  for  that  operational  definition.  For  example,  if  all 
of  the  maximum  possible  rating  proportions  (based  on  the 
original  rating  plus  the  standard  error)  were  used  in  equa¬ 
tion  (5),  the  ACR  value  might  have  been  represented  as 


follows: 


ACR  -  0.952[ (0.352  +  0.352  +  0.352)/3]  -  0.335 


In  the  same  fashion,  using  the  lowest  possible  rating 
proportions  to  calculate  the  ACR  value  would  give  a  figure 
of  0.031.  If  the  uncontrolled  sampling  error  assumption  was 
not  made  in  the  last  chapter,  the  IP  value  calculated  above 
should  have  been  reported  as  0.16  (+  0.1750/  -  0.129). 
These  figures  indicate  the  conditions  associated  with  the 
ACR  calculation  reported  in  the  last  section  of  this  chapter 
would  probably  occur  16%  of  the  time,  but  possibly  as  often 
as  33%,  or  as  infrequently  as  3%  of  the  time.  This  degree 
of  inaccuracy  is  generally  unacceptable  in  an  empirical 


As  a  result  of  the  calculations  of  the  actual  sampling 
errors  associated  with  the  final  data  reductions  reported  in 
the  last  section,  the  assumption  regarding  the  uncontrolled 
sampling  error  is  considered  justified.  Because  of  the 
potential  variation  in  the  data  demonstrated  in  the  last 
paragraph,  the  decision  to  evaluate  the  second  research 
question  in  the  last  section  was  also  sound.  Based  on  the 
potential  variation  in  the  calculated  IP  value,  it  is  also 
possible  to  state  that  improved  productivity  might  have  been 
validated  in  many  more  facilities,  given  a  larger  sample 
size. 

As  noted  earlier  in  this  chapter,  one  plant  representa¬ 
tive  office  did  not  permit  the  requested  on-site  interview. 
While  this  was  not  considered  as  significant  a  deviation  as 
the  sampling  error  problem  just  described,  it  should  be 
acknowledged  as  a  potential  non  response  error.  In  a  re¬ 
search  study  with  a  limited  sample  size  such  as  this,  the 
problem  created  by  non  response  error  occurs  in  the  area  of 
generalizability  of  the  data.  Although  no  statistical  tests 
are  available  to  counteract  this  problem,  a  solution  will  be 
proposed  in  Chapter  V.  Because  of  the  assumptions  and 
limitations  described  in  Chapter  III,  no  other  statistical 
tests  are  appropriate  for  the  data  derived  above. 


Overview 


Chapter  I  introduced  the  general  issue  —  the  reduction 
of  costs  in  weapon  system  acquisitions  —  behind  the  study 
just  reported.  The  research  questions  and  objectives  of 
this  study  were  also  introduced  in  Chapter  I.  The  ultimate 
objective  of  this  study  was  an  evaluation  of  the  relation¬ 
ships  between  quality  assurance  process  controls,  productiv¬ 
ity  and  weapon  system  costs.  Chapter  II  described  the 
relevant  literature  to  support  the  objectives  established  in 
Chapter  I.  In  analyzing  that  literature,  it  was  determined 
there  was  a  serious  flaw  in  the  current  data  base,  in  that 
there  was  little  empirical  evidence  relating  process  con¬ 
trols  to  productivity  and  cost. 

As  a  result  of  the  shortcomings  in  the  current  litera¬ 
ture,  a  descriptive  study  was  undertaken  to  develop  empiri¬ 
cal  data  relative  to  the  research  objectives.  Chapter  III 
described  the  detailed  survey  instrument  used  to  gather  the 
descriptive  data  regarding  process  controls,  productivity 
and  cost.  The  results  of  these  in-depth  surveys  were  repor¬ 
ted  in  Chapter  IV,  and  then  analyzed  in  terms  of  the  origi¬ 
nal  investigative  and  research  questions.  Based  upon  that 
analysis,  it  was  determined  that  productivity  improvements 
(and  possibly  associated  weapon  system  costs),  resulting 
from  adequate  process  and  cost  controls,  were  not  supported. 


The  reason  the  relationships  in  question  were  not  sup¬ 
ported  was  inadequate  government  validation  of  contractor 
claims  for  improved  productivity  and  associated  cost  reduc¬ 
tions.  The  significance  of  this  finding  will  be  examined 
further  in  the  next  section  of  this  chapter.  A  third  sec¬ 
tion  of  this  chapter  will  then  discuss  the  practical  impli¬ 
cations  of  this  study.  These  implications  will  be  related 
back  to  issues  previously  identified  in  Chapters  II,  III  and 
IV.  Finally,  this  chapter  concludes  with  this  writer's 
recommendations  for  follow-on  or  revised  studies  based  on 
the  data  generated  by  this  report. 

Significance  of  the  Findings 

The  research  described  in  the  preceding  chapters  was 
designed  to  gather  data  for  the  development  of  hypotheses  to 
test  whether  process  controls  are  related  to  productivity 
and  weapon  system  costs.  This  approach  was  selected  because 
the  present  literature  presented  no  empirical  evidence  re¬ 
lating  the  elements  just  listed,  and  because  no  evidence  was 
presented  to  demonstrate  that  cost  savings  resulting  from 
such  relationships  were  passed  on  to  DoD.  This  study  was 
only  partially  successful  in  terms  of  these  design  goals. 

The  first  shortcoming  in  the  present  literature  —  the 
lack  of  empirical  data  relating  process  controls  to  produc¬ 
tivity  and  cost  —  resulted  from  inadequate  government  test¬ 
ing  of  the  elements  of  process  control  described  in  this 
study.  This  study  did  overcome  one  aspect  of  that  shortcom- 
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ing  by  providing  a  methodology  to  gather  relevant  data 
concerning  the  relationships  that  might  exist.  The  failure 
of  this  research  effort  to  support  the  relationships  claimed 
in  the  literature  was  not  a  fault  of  the  study  design.  As 
noted  in  Chapter  IV,  there  is  some  evidence  to  show  that  the 
issues  under  investigation  are  related.  However,  the  pre¬ 
sent  DoD  surveillance  approach  does  not  provide  a  method  to 
validate  contractor  claims  regarding  productivity  improve¬ 
ments  which  occur  because  of  adequate  process  or  cost  con¬ 
trols.  The  implications  of  this  finding  will  be  discussed 
in  the  next  section.  For  now,  it  can  be  stated  there  is  a 
way  to  address  the  first  shortcoming  in  the  present  litera¬ 
ture  through  use  of  an  in-depth  survey  such  as  this,  and 
through  incorporation  of  the  recommended  surveillance  method 
changes  described  below. 

The  second  shortcoming  in  the  current  literature  —  the 
lack  of  evidence  to  demonstrate  cost  savings  are  passed  on 
to  DoD  --  was  also  partially  resolved  by  this  study.  The 
methodology  used  provides  a  way  to  determine  if  cost  savings 
are  occurring  as  a  result  of  adequate  process  controls  and 
related  productivity  improvements.  As  above,  the  failure  of 
the  government  in-plant  personnel  to  validate  that  savings 
are  passed  on  to  DoD  can  be  overcome  through  changes  in  the 
government  surveillance  methods.  But,  there  is  now  a  method 
available  to  overcome  the  second  shortcoming  in  the  present 
literature . 
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As  a  result  of  the  partial  success  in  overcoming  the 
shortcomings  of  the  present  literature,  this  study  does 
provide  a  base  on  which  to  build  future  studies.  The  recom¬ 
mended  follow-on  studies  will  be  addressed  in  the  last 
section  of  this  chapter.  This  study  does  add  to  the  empiri¬ 
cal  data  available  to  DoD,  and  has  actually  built  upon  that 
data  provided  in  earlier  empirical  studies.  Various  ele¬ 
ments  of  process  control  identified  in  the  previous  studies 
were  incorporated  into  the  interview  checklist  used  in  this 
study.  For  example,  the  use  of  capital  equipment  in  produc¬ 
ing  better  quality  products  (18:82)  was  examined,  as  was  the 
effect  of  equipment  set-up  (15:45-46).  In  addition,  two  of 
the  previous  studies  (2;  27)  addressed  the  role  of  scrap  and 
rework  in  productivity  improvements,  and  these  elements  of 
process  control  were  evaluated  by  the  instant  study. 

Practical  Implications  of  the  Results 

Three  major  recommendations  can  be  made  to  the  DoD 
community  as  a  result  of  the  findings  presented  in  this 
research  report.  The  first  recommendation  results  from  the 
discussion  in  the  last  section  regarding  the  surveillance 
methods  used  by  DoD  in-plant  personnel.  The  second  recom¬ 
mendation  addresses  the  issue  of  MIL-Q-9858A  cost  data  pre¬ 
paration  first  identified  in  Chapter  II.  The  third  recom¬ 
mendation  is  based  on  observations  made  during  the  inter¬ 
views  reported  in  Chapter  IV  regarding  the  role  of  Tech  Mod 
and  Man  Tech  funding  incentives. 
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DoD  Surveillance  Methods.  As  noted  in  the  preceding 
section,  as  well  as  when  examining  the  relationships  in 
terms  of  the  original  research  questions  in  Chapter  IV, 
process  controls  could  not  be  judged  positively  related  to 
improved  productivity  or  reduced  weapon  system  costs  because 
of  inadequate  data  validation  by  DoD  in-plant  personnel. 
This  apparent  weakness  in  the  DoD  surveillance  approach 
should  be  examined  by  all  the  military  departments  respons¬ 
ible  for  major  weapon  system  procurements.  Two  different 
approaches  are  recommended  in  this  regard.  The  first  change 
requires  coordination  and  improvement  in  the  methods  of 
assessing  the  adequacy  of  contractor  process  and  quality 
cost  controls.  The  second  change  is  related  to  the  pricing 
and  cost  evaluations  performed  by  the  in-plant  personnel. 

In  terms  of  assessing  the  adequacy  of  contractor  pro¬ 
cess  and  quality  cost  controls,  a  joint  service  manual  has 
been  prepared  by  the  Defense  Logistics  Agency  (DLA).  DLAM 
8200.1,  Procurement  Quality  Assurance  (9) ,  describes  the 
basic  surveillance  approach  to  analyzing  contractor  quality 
systems  and  programs.  DLA  and  the  other  military  depart¬ 
ments  have  supplemented  this  basic  manual  with  other  docu¬ 
ments  and  regulations  providing  additional  guidance  on  moni¬ 
toring  the  adequacy  of  contractor  controls.  For  example, 
the  Air  Force  has  issued  AFCMDR  178-1,  Contractor  Management 
System  Evaluation  Program  (CMSEP),  which  addresses  quality 
assurance  (QA)  and  other  contractually  required  management 


systems  (10).  The  QA  portion  of  the  CMSEP  manual  is  based 
on  H50,  the  previously  referenced  document  (30)  for  asses¬ 
sing  contractor  quality  programs  Implemented  in  accordance 
with  MIL-Q-9858A.  There  are  numerous  other  QA  manuals  used 
by  DLA,  the  Air  Force,  the  Army  and  the  Navy,  but  a  review 
of  all  such  manuals  is  beyond  the  scope  of  this  study. 

While  all  of  the  DoD  QA  manuals  have  some  overlap  in 
the  area  of  in-plant  surveillance,  those  referenced  thus  far 
do  not  require  validation  of  trend  or  cost  data  generated  by 
the  contractor  QA  department.  In  most  facilities  visited 
during  the  course  of  this  study,  the  DoD  in-plant  personnel 
only  verified  that  the  contractor  maintained  the  required 
data.  If  DoD  ever  hopes  to  achieve  the  level  of  cost  con¬ 
trol  called  for  by  former  Deputy  Secretary  of  Defense  Frank 
Carlucci  (5),  it  is  recommended  that  the  various  QA  surveil¬ 
lance  manuals  be  revised  to  require  validation  of  the  accu¬ 
racy  of  the  contractor  data. 

In  terms  of  the  second  surveillance  change  recommended 
above,  the  military  services  need  to  expand  their  pricing 
and  cost  evaluations  to  assure  assessment  of  productivity 
and  cost  improvements  which  result  from  adequate  process 
controls.  In  addition  to  those  military  departments  men¬ 
tioned  above,  the  Defense  Contract  Audit  Agency  (DCAA) 
should  also  be  brought  into  this  expanded  surveillance  ap¬ 
proach.  It  is  thus  recommended  that  DoD  cost  and  pricing 


personnel  be  made  avare  of  the  potential  for  cost  savings 
that  might  occur  in  the  contractor  manufacturing  and  fabri¬ 
cation  activities.  Where  productivity  and  cost  improvements 
have  been  validated  by  DoD  QA  personnel,  such  improvements 
should  be  considered  in  follow-on  procurement  and  overhead 
negotiations. 

MIL-Q-9858A  Cost  Data.  Closely  related  to  the  proposed 
changes  in  DoD  surveillance  methods  is  the  need  for  changes 
in  the  MIL-Q-9858A  requirements  for  development  of  contrac¬ 
tor  QA  cost  data.  While  this  study  confirmed  that  the 
contractors  do  indeed  maintain  the  required  data,  each  of 
the  contractors  used  a  different  format  and  approach  to  the 
collection  of  the  data.  This  fact  is  unfortunately  permit¬ 
ted  by  the  language  of  MIL-Q-9858A,  which  states,  "The 
specific  quality  cost  data  to  be  maintained  and  used  will  be 
determined  by  the  contractor  [39:3]."  If  DoD  is  to  be 
successful  in  implementing  the  improved  surveillance  methods 
described  in  the  last  subsection,  the  DoD  in-plant  personnel 
responsible  for  evaluating  the  quality  cost  data  need  a 
uniform  format  and  level  of  indenture  to  assure  equal  stan¬ 
dards  exist  for  the  assessment  and  validation  of  such  data. 

It  is  recommended  that  DoD  address  this  issue  by  con¬ 
vening  a  joint  military  and  industry  review  panel  to  develop 
revised  standards  for  the  collection  of  quality  cost  data. 
Such  data  should  not  just  be  made  available  for  on-site 


government  review,  as  is  now  the  case,  but  should  be  made  a 


deliverable  data  item  in  all  major  weapon  system  procure¬ 
ments.  It  is  recognized  that  such  data  must  be  treated 
confidentially  because  of  its  proprietary  nature,  but  such 
constraints  already  exist  for  much  of  the  cost  and  pricing 
data  now  processed  by  DoD.  Once  the  data  is  made  available, 
DoD  cost  and  pricing  personnel  should  review  it  to  validate 
savings  accrued  as  a  result  of  adequate  process  controls  and 
improved  productivity  are  passed  on  to  DoD. 

Tech  Mod  and  Man  Tech  Incentives.  The  final  practical 
implication  of  the  data  generated  by  this  study  results  from 
observations  made  in  examining  the  use  of  capital  equipment 
in  the  control  of  the  processes.  While  most  of  the  facili¬ 
ties  visited  had  some  new  capital  equipment  which  apparently 
contributed  to  the  control  of  the  processes,  there  were  no 
contractual  incentives  provided  the  contractor  to  relate  the 
use  of  such  equipment  to  improved  productivity.  As  pre¬ 
viously  noted,  such  incentives  are  recommended  in  the  Air 
Force  guide  to  technology  modernization  (12:121). 

It  is  recommended  that  the  Air  Force  and  other  services 
reexamine  the  Tech  Mod  and  Man  Tech  contracts  to  ensure  use 
of  contractual  incentives  based  on  increased  productivity. 
Implementation  of  this  recommendation  would  provide  two 
benefits  to  DoD.  First,  it  would  ensure  that  some  of  the 
elements  of  process  control  evaluated  in  this  study  are 
assessed  and  validated  by  DoD  cost  and  pricing  personnel. 
Secondly,  such  a  review  could  consider  other  factors  asso- 


elated  with  Tech  Mod  but  not  presently  addressed  in  the  Air 
Force  guide,  such  as  the  direct  and  indirect  labor  and 
material  cost  reviewed  by  this  study.  The  previously  re¬ 
viewed  empirical  study  by  Berry  and  Bland  (2:90)  proposed  a 
model  which  might  be  considered  in  the  evaluation  of  this 


recommendation . 


Recomme 


In  line  with  the  practical  implications  of  the  findings 
discussed  in  the  last  section,  a  number  of  studies  could  be 
initiated  by  DoD.  The  first  such  study  recommended  would  be 
a  review  of  the  various  DoD  quality  assurance  and  pricing 
manuals,  directives  and  regulations  to  develop  a  uniform 
approach  to  validating  and  using  contractor  data  on  process 
and  productivity  trends.  Such  a  study  could  provide  the 
tools  to  assure  cost  savings  resulting  from  adequate  process 
controls  are  indeed  passed  on  to  DoD.  A  second  study  in 
this  area  could  focus  on  the  necessary  cost  elements  to  be 
reported  by  contractors  responsible  for  developing  QA  cost 
controls  in  accordance  with  MIL-Q-9858A.  The  results  of 
such  a  study  could  be  presented  to  the  previously  recommen¬ 
ded  joint  review  panel.  A  third  study,  to  close  the  loop 
addressed  by  the  practical  implications  discussed  above, 
might  address  the  elements  of  capital  equipment  productivity 
measures  to  be  considered  in  Tech  Mod  and  Man  Tech  incentive 
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In  addition  to  addressing  the  implications  of  this 
study,  future  research  should  be  considered  to  build  upon 
the  theoretical  base  of  knowledge  regarding  the  relation¬ 
ships  among  procesa  controls,  productivity  and  weapon  system 
costs.  Once  better  DoD  surveillance  methods  are  developed, 
a  follow-on  study  could  be  conducted  to  see  if  the  relation¬ 
ships  described  in  Chapter  IV  are  marginally  improved.  A 
number  of  methodological  improvements  could  be  made  by  ex¬ 
panding  the  scope  of  this  study.  For  example,  all  major 
weapon  system  contractors  listed  in  DoD  4105. 59-H  could  be 
included  by  developing  additional  process  control  question¬ 
naires  based  on  a  variety  of  processes.  The  operational 
definitions  presented  in  Chapter  IV  are  not  peculiar  to  the 
processes  evaluated  by  this  study. 

This  proposed  follow-on  study  could  be  executed  in  a 
couple  of  different  ways.  For  example,  all  of  the  plant 
representative  offices  cognizant  of  major  weapon  system 
contractors  could  be  visited  or  assessed  through  mail  ques¬ 
tionnaires,  or  a  larger  sample  could  be  drawn  that  would 
ensure  coverage  of  plant  representative  offices  from  all  the 
services.  This  approach  would  overcome  the  concern  for  the 
generalizability  of  the  data  developed  by  the  instant  study 
by  demonstrating  whether  the  relationships  in  question  hold 
for  Army  and  Navy  plant  representative  offices  (ARPROs  and 
NAVPROs),  as  well  as  other  Air  Force  and  DCAS  plant  repre¬ 
sentative  offices  (AFPROs  and  DCASPROs)  not  covered  by  this 
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study*  Such  an  expanded  study  would  also  eliminate  the 
uncontrolled  sampling  error  constraint  placed  on  this  study 
by  the  small  sample  size.  However,  problems  with  non  re¬ 
sponse  errors,  identified  in  the  last  chapter  (13:299-301), 
would  become  more  critical  to  an  expanded  study. 

An  expanded  study  based  on  the  data  generated  by  this 
report  could  increase  the  reliability  of  the  conclusions 
developed  above.  One  way  to  accomplish  this  objective  would 
be  to  pre-test  the  interview  instrument  along  the  lines 
suggested  in  Chapter  III.  In  addition,  the  normal  statisti¬ 
cal  tests  of  nominal  data,  such  as  the  binomial  and  (2  tests 
described  in  Chapter  III,  could  be  conducted.  This  approach 
would  provide  a  measure  of  the  statistical  significance  of 
the  data  generated.  But,  it  would  also  require  satisfactory 
data  regarding  positive  relationships  among  the  elements 
studied,  so  that  null  and  alternate  hypotheses  could  be 
tested  based  on  a  specific  set  of  decision  rules. 

Finally,  other  follow-on  research  might  consider  ele¬ 
ments  of  productivity  not  addressed  by  this  report.  For 
example,  the  Tuttle  report  (32)  mentioned  in  Chapter  II, 
along  with  a  recently  released  productivity  taxonomy  deve¬ 
loped  by  Sink,  et  al  (30),  could  be  used  to  identify  addi¬ 
tional  elements  of  productivity  for  evaluation  in  a  more 
encompassing  follow-on  study.  The  Committee  on  Naval  Ship¬ 
building  Technology  (7:28,47)  recommended  consideration  of 
such  factors  as  management  techniques,  training,  planning, 


scheduling,  and  sequencing  of  the  work  operations  to  improve 
the  processes.  According  to  Hanifin  (17:53-54),  the  effect 
of  design  and  computer  integrated  manufacturing  systems  also 
have  significant  impact  on  quality  and  productivity.  While 
all  of  the  elements  affecting  product  quality  and  producti¬ 
vity  may  be  impractical  for  a  single  study,  smaller  studies 
can  be  used  to  build  toward  a  larger,  all-encompassing  model 
of  the  relationships  among  process  controls,  productivity 
and  weapon  system  costs.  This  study  has  already  contributed 
to  such  a  model  by  providing  a  methodology  and  empirical 
data  base  upon  which  to  build  the  recommended  follow-on 
studies . 
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Evaluation  of  Production  and  Manufacturing  Processes 

General  Issues : 

1.  What  are  the  major  contract  quality  requirements 
for  the  products  being  manufactured  at  this  facility? 

2.  What  is  the  basic  DoD  manual  you  use  to  develop 
your  surveillance  of  the  contractor's  process  controls? 

3.  What  type  of  surveillance  does  the  contractor 
perform  of  his  machining  and  heat  treat  operations? 

4.  What  are  the  quality  assurance  criteria  that  you 
use  to  evaluate  the  contractor's  machining  operations  and 
special  processes? 

5.  Have  you  ever  considered  evaluation  of  1st  piece, 
in-process,  tooling  used  as  a  media  of  inspection,  further 
assembly,  final  inspection,  NDT/NDI,  or  product-oriented 
procedures  evaluations  (POPE)  (select  the  appropriate  cate¬ 
gories  not  mentioned  in  the  preceding  answer]  to  determine 
the  adequacy  of  the  contractor's  control  of  his  machining 
operations  and  special  processes? 

6.  How  do  you  evaluate  the  equipment  capability  (ac¬ 
curacy  and  precision)  and  qualification  for  machining  ope¬ 
rations  and  special  processes? 

7.  How  do  you  evaluate  special  operating  environment 
controls  such  as  cleanliness,  temperature  and  humidity  in 
the  manufacturing  area? 

8.  How  do  you  evaluate  the  contractor's  compliance 
with  his  work  instructions,  routing  sheets,  specifications, 
drawings  and  workmanship  standards  in  the  manufacturing  and 
special  process  areas? 


9.  How  would  you  define  process  control? 

10.  How  would  you  define  process  yield? 


Machining  Operations: 

11.  Vhat  types  of  basic  machining  operations  does  the 
contractor  perform  in-house  (e.g.,  milling,  drilling,  form¬ 
ing, etc.  )? 

12.  What  type  of  analysis  do  you  perform  of  the 
contractor's  equipment  maintenance,  including  such  actions 
as  trueing  the  ways  and  adjusting  for  spindle  wear? 

13.  How  do  you  verify  that  the  contractor  performs 
the  required  calibration  on  his  machines? 

14.  How  does  the  contractor  control  for  wear  of  his 
tooling  and  fixtures? 

15.  What  do  you  do  to  verify  the  contractor's  quali¬ 
fication  and  calibration  of  his  machine  tools  and  fixtures, 
especially  those  used  as  a  media  of  inspection? 

16.  What  type  of  evaluation  do  you  perform  of  the 
lubricants  and  cooling  oils  (in  terms  of  specif ication  re¬ 
quirements  or  operating  instructions)  used  during  machining 
operations? 

17.  How  do  you  assure  that  the  machine  operator  uses 
the  proper  set-up,  and  maintains  the  planned  feeds  and 
speeds  during  production? 

18.  What  type  of  verification  do  you  perform  of  the 
operator's  changing  of  the  tooling  inserts  and  compensating 
machine  offsets? 

19.  How  do  you  evaluate  the  operator's  depth  of  cut 
(rough  versus  finish),  and  whether  the  contractor  controls 
this  activity  to  prevent  unwanted  work  hardening  and  theimal 
growth? 

20.  If  the  contractor  uses  statistical  analysis  of 
his  machining  operations,  what  do  you  do  to  evaluate  these 
controls? 


21.  Have  you  ever  considered  evaluation  of  the  con¬ 
tractor's  x-bar  and  R  charts,  the  number  of  standard  devia¬ 
tions  used  in  conjunction  with  such  charts,  or  his  analysis 
of  the  process  variability,  including  shifting  of  the  pro¬ 
cess  average  [select  the  appropriate  categories  not  men¬ 
tioned  in  the  previous  answer]? 
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22.  What  type  of  advanced  machining  techniques  does 
the  contractor  use,  such  as  CAD/CAM,  numerical  control, 
robotics  or  Integrated  gaging  (for  automatic  size  control 
and  cutting  tool  compensation)? 

23.  How  do  you  evaluate  the  contractor's  control 
(e.g.,  development  and  use  of  N/C  tapes,  operator  training, 
etc.)  of  these  advanced  techniques  described  in  the  response 
to  the  preceding  question? 

Heat  Treating  Operations : 

24.  What  type  of  heat  treating  does  the  contractor 
perform? 

25.  How  do  you  evaluate  the  contractor's  preparations 
prior  to  the  heat  treat  process,  such  as  surface  cleanliness 
and  eddy  current  testing  (for  electrical  conductivity  and 
location  in  the  rack)? 

26.  How  do  you  verify  the  rate  of  heating  and  soak 

time? 

27.  What  type  of  analysis  do  you  make  of  the  quenching 
operation,  including  solution  temperature  and  circulation, 
timeliness  and  velocity  of  immersion,  and  use  of  appropriate 
rinses? 

28.  How  do  you  evaluate  the  cooling  rate  when  the 
contractor's  procedures  call  for  annealing? 

29.  What  type  of  testing  does  the  contractor  perform 
to  verify  the  material  properties  after  heat  treating? 

30.  How  do  you  evaluate  the  adequacy  of  the  contrac¬ 
tor's  tests  described  in  the  previous  response? 

31.  Have  you  ever  considered  evaluating  properties 
such  as  intergranular  corrosion,  eutectic  melting,  oxida¬ 
tion,  decarburization,  and  nitriding  [select  appropriate 
properties  not  mentioned  in  the  response  to  the  previous  two 
questions],  as  required  by  such  specifications  as  MIL-H- 
6088F  and  MIL-H-6875G? 


Evaluation  of  Contractor  Cost  Controls 


32. 
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34.  What  is  the  government  role  in  evaluating  contrac¬ 
tor  rework,  repair  and  scrap? 

35.  What  type  of  trend  analysis  does  the  contractor 
perform  on  rework,  repair,  scrap  and  reject  rates? 

36.  How  do  you  assess  the  contractor's  trend  data? 

37.  What  role  does  improved  product  quality  play  in 
allowing  the  contractor  to  use  sampling  in  lieu  of  100% 
inspection,  or  reducing  the  size  of  the  inspection  samples? 

38.  How  do  you  evaluate  the  contractor's  cost  data 
required  by  MIL-Q-9858A,  especially  in  the  areas  of  preven¬ 
tion  and  correction  of  nonconforming  supplies? 

39.  In  addition  to  scrap,  rework  and  repair  costs,  how 
do  you  evaluate  the  contractor's  overall  discrepancy  costs, 
(including  such  expenses  as  replacement  and  retest  of  defec¬ 
tive  units  and  the  cost  of  engineering  changes  generated  as 
a  result  of  nonconforming  material)? 


Relationships  Among  Process  Controls.  Cost  and  Productivity 

40.  What  contractual  funding  does  the  contractor  re¬ 
ceive  for  Manufacturing  Technology  or  Tech  Mod? 

41.  How  does  the  contractor  use  capital  equipment  to 
improve  his  process  controls? 

42.  What  type  of  corrective  action  does  the  contractor 
typically  perform  to  reduce  the  causes  of  nonconforming 
material  in  the  heat  treat  or  machining  departments? 

43.  What  effect  does  corrective  action  have  on  reduc¬ 
ing  the  amount  of  defective  product  in  the  machine  shop  or 
heat  treat  departments? 

44.  How  would  you  relate  the  contractor's  efforts  to 
reduce  the  amount  and  causes  of  nonconforming  material  to 
improved  productivity  or  yield? 

45.  What  effect  do  you  think  the  contractor's  rework 
and  repair  efforts  have  on  productivity? 

46.  Using  the  criteria  from  the  preceding  two  ques¬ 
tions,  what  are  the  productivity  trends  in  the  machining  and 
heat  treat  areas? 


47.  What  effect  does  the  contractor's  efforts  to  im¬ 
prove  his  process  controls  and  productivity  ultimately  have 
on  his  costs? 


Evaluation  of  Supporting  Data 

48.  What  types  of  contractual  incentives  are  provided 
to  the  contractor  to  improve  productivity? 

49.  What  types  of  cost  or  trend  data  does  the  contrac¬ 
tor  provide  for  in-house  government  review? 

50.  What  types  of  cost  or  trend  data  is  the  contractor 
required  to  submit  to  the  government  as  deliverable  data  in 
accordance  with  a  DD  Form  1423? 

51.  How  do  you  determine  the  validity  or  accuracy  of 
the  contractor  data  that  is  delivered  to  the  government  or 
made  available  for  on-site  review? 

52.  How  is  the  contractor  trend  or  cost  data  from  the 
machining  and  heat  treat  departments  used  to  determine  whet¬ 
her  productivity  improvements  result  in  a  reduced  cost  to 
the  government? 

53.  How  does  the  government  assess  the  effect  of  re¬ 
lated  cost  savings  (such  as  reduced  direct  and  indirect 
labor  or  overhead  costs)  which  may  have  resulted  from  im¬ 
proved  process  controls  or  productivity? 


Respondent  Identification 

54.  What  is  your  name? 

55.  What  is  your  organization? 

56.  What  is  your  title  and  position  within  the  organi¬ 
zation? 


57.  How  long  have  you  held  your  current  position? 

58.  What  prior  experience  do  you  have  in  the  subject 
areas  discussed  above? 


fill. 


Appendix  B:  Study  Population  Listing 


# 

FACILITY 

LOCATION 

01 

DCASPRO  Hayes  Int. 

Dothan,  AL 

02 

DCASPRO  Grumman  Aerospace 

Stuart,  FL 

03 

DCASPRO  AVCO  Lycoming  Div. 

Stratford,  CT 

04 

DCASPRO  Sundstrand 

Rockford,  IL 

05 

DCASPRO  Teledyne  CAE 

Toledo,  OH 

06 

DCASPRO  Williams  Int. 

Walled  Lake,  MI 

07 

DCASPRO  Goodyear  Aerospace 

Akron,  OH 

08 

DCASPRO  McDonnell/Rockwell 

Tulsa,  OK 

09 

DCASPRO  General  Dynamics 

San  Diego,  CA 

10 

DCASPRO  Ford  Aeronutronic 

Newport  Beach,  CA 

11 

DCASPRO  Westinghouse 

Sunnyvale,  CA 

12 

DCASPRO  McDonnell  Douglas 

Huntington  Beach,  CA 

13 

ARPRO  Textron  (Bell  Helicopter) 

Fort  Worth,  TX 

14 

ARPRO  Hughes  Helicopter,  Inc. 

Culver  City,  CA 

15 

ARPRO  Boeing-Vertol 

Philadelphia,  PA 

16 

NAVPRO  Grumman  Aerospace 

Bethpage,  NY 

17 

NAVPRO  Vought  Corp. 

Dallas,  TX 

18 

NAVPRO  GE  (Aircraft  Engine) 

Lynn,  MA 

19 

NAVPRO  Lockheed  California 

Burbank,  CA 

20 

NAVPRO  McDonnell  Douglas 

St.  Louis,  MO 

21 

NAVPRO  General  Dynamics 

Pomona,  CA 

22 

NAVPRO  United  Technologies 

Stratford,  CT 

23 

AFPRO  Boeing 

Seattle,  WA 

24 

AFPRO  Hughes  Aircraft 

Los  Angeles,  CA 

25 

AFPRO  Rockwell  International 

Anaheim,  CA 

26 

AFPRO  Rockwell  International 

Los  Angeles,  CA 

27 

AFPRO  Rockwell  Int.  (Rocketdyne) 

Canoga  Park,  CA 

28 

AFPRO  Northrop  Corp. 

Hawthorne,  CA 

29 

AFPRO  Martin-Marietta 

Denver,  CO 

30 

AFPRO  Douglas  Aircraft 

Long  Beach,  CA 

31 

AFPRO  Lockheed-Georgia 

Marietta,  GA 

32 

AFPRO  Fairchild  Republic 

Farmingdale,  NY 

33 

AFPRO  Hughes  Missiles 

Tuscon,  AZ 

34 

AFPRO  Thiokol  Corp. 

Brigham  City,  UT 

35 

AFPRO  GE  Aircraft  Engine 

Cincinnati,  OH 

36 

AFPRO  GD,  Fort  Worth  Div. 

Fort  Worth,  TX 

37 

AFPRO  Westinghouse  Electric 

Baltimore,  MD 

38 

AFPRO  Pratt  &  Whitney 

West  Palm  Beach,  FL 

39 

AFPRO  Pratt  &  Whitney 

East  Hartford,  CT 

40 

AFPRO  Lockheed  Missile 

Sunnyvale,  CA 

41 

AFPRO  CSD  Division 

Sunnyvale,  CA 

42 

AFPRO  Rockwell  International 

Columbus,  OH 

43 

AFPRO  Boeing  Military  Airplane 

Wichita,  KS 

Appendix  D:  Saeple  Introductory  Letter 

21  May  1984 

REPLY  TO: 

SUBJECT:  Plant  Visit  to  Gather  Thesis  Data 

TO:  Commander 

AFPRO  - 


1.  Reference:  Telecon  between  Mr.  -  and  Mr.  M.  O'Meara, 

dated  18  May  84,  same  subject. 

2.  Request  permission  for  Mr.  Michael  V.  O'Meara  to  visit 
your  offices  to  collect  data  for  an  Air  Force  sponsored 
Master's  Thesis.  Mr.  O'Meara's  research  is  designed  to 
evaluate  the  relationship  between  process  controls,  produc¬ 
tivity  and  weapon  system  costs.  The  data  he  proposes  to 
collect  will  be  gathered  by  on-site  interviews  and  record 
reviews.  He  will  need  to  interface  briefly  with  QA  Super¬ 
visory  and  Technical  Specialists,  and  Contracting  personnel 
familiar  with  the  contractor's  financial  proposals.  It  is 
estimated  that  Mr.  O'Meara  can  gather  the  necessary  data  in 
four  to  six  hours.  All  data  regarding  the  facility,  pro¬ 
grams  and  personnel  interviewed  will  be  kept  strictly  confi¬ 
dential,  and  the  final  report  will  be  based  on  summarized 
results  from  all  facilities  visited. 

3.  Mr.  O'Meara  has  made  preliminary  contact  with  Mr. - . 

They  have  tentatively  agreed  to  a  visit  date  of  20  Jun  84. 
With  your  permission,  Mr.  O'Meara  will  reconfirm  this  date 
within  the  next  two  weeks.  Required  security  clearances 
will  be  forwarded  prior  to  the  visit.  Mr.  O'Meara  has  a 
-  security  clearance. 


Commander 
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